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Abstract 


This  thesis  examines  the  use  of  decision  support  system  (DSS)  technology  within 
the  U.S.  Air  Force’s  contracting  source  selection  process.  DSS  technology  has  been 
shown  to  be  an  effective  aid  to  complex  decision  processes  such  as  source  selection 
decisions  and  the  possible  effects  of  utilizing  this  technology  are  the  focus  of  this  study. 
Conclusions  drawn  from  this  thesis  may  guide  the  contracting  community  to  readdress 
the  possibility  of  implementing  DSS  technology  within  the  source  selection  process. 

This  study  evaluated  statistical  data  from  a  previous  experiment  to  determine  the 
effect  of  DSS  use  on  time,  confidence,  and  decision  quality.  This  study  also  included 
interviews  of  individuals  involved  in  the  source  selection  process  to  determine  their 
perceptions  of  DSS  technology.  It  was  found  that  the  perception  of  DSS  technology 
within  the  source  selection  community  has  improved  within  the  last  five  years.  It  was 
also  found  that  source  selection  experts  foresee  DSS  implementation  within  the  source 
selection  process  within  the  near  future.  In  fact,  the  source  selection  experts 
interviewed  believed  that  group  DSS  would  provide  the  greatest  benefit  to  the  process. 
Possible  explanations  and  implications  of  this  discovery  are  provided. 
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COMPUTER-BASED  DECISION  SUPPORT  SYSTEM 
USE  IN  CONTRACTING'S 
SOURCE  SELECTION  PROCESS 


I.  Introduction 


General  Issue 

The  ability  to  make  optimal  decisions  is  an  important  aspect  of  all  human 
existence.  Whether  it  be  the  decision  of  what  to  plant  in  the  fields,  what  type  of 
insurance  to  buy,  or  what  technological  applications  to  purchase,  decisions  represent 
the  important  choices  in  life.  The  decision  making  process  can  be  defined  as  the  steps 
the  decision  maker  goes  through  in  preparation  of  reaching  a  decision. 

Obviously,  the  decision  process  has  a  great  impact  of  the  eventual  decision.  For 
example,  if  the  decision  scenario  was  to  choose  what  car  insurance  to  purchase,  there 
would  be  several  decision-making  processes  that  could  be  utilized.  The  decision-maker 
could  select  the  insurance  policy  with  the  nicest  brochure  or  could  follow  a  systematic 
and  complete  decision  process.  Which  decision  process  would  you  prefer  using  to 
select  your  insurance  coverage?  How  we  reach  a  decision  greatly  impacts  the  ultimate 
decision  reached  so  we  should  strive  to  improve  how  decisions  are  made  in  order  to 
improve  the  resultant  decision. 

Since  the  decision  making  process  is  so  vital  to  better  decisions,  it  is  not 
surprising  that  a  significant  amount  of  academic  literature  has  been  developed 
addressing  this  area.  Herbert  Simon  is  widely  considered  to  be  the  grandfather  of 
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decision  theory  and  is  considered  the  most  well  known  and  accepted  of  the  decision 
theorists.  He  has  created  an  increased  understanding  of  the  decision  process  and  how 
to  improve  it. 

The  emphasis  on  decision  making  and  decision  theory  has  also  progressed  from 
the  academic  to  the  business  community.  At  an  organizational  level,  improving  decision 
making  is  seen  as  a  way  to  increase  competitiveness  and  improve  the  profit  margin. 
Because  of  the  perceived  monetary  benefit  of  improved  decision  making,  significant 
investments  into  this  field  have  been  made. 

The  Department  of  Defense  (DoD)  has  also  pursued  improved  decision  making 
through  a  variety  of  methods.  A  fairly  recent  development  In  the  decision-making  arena 
is  the  usage  of  computers  to  aid  in  the  decision  process.  There  have  been,  in  recent 
years,  developments  of  computer-based  decision  aids  that  facilitate  complex  decision¬ 
making.  This  study  will  address  the  applicability  of  this  technology  in  a  specific  decision 
making  process  within  the  Department  of  the  Air  Force,  namely  the  source  selection 
process. 

The  source  selection  process  is  a  process  by  which  the  United  States  Air  Force 
chooses  between  alternative  suppliers  to  perform  a  service  or,  most  commonly,  to 
provide  a  product  or  weapon  system.  The  individual  who  has  the  responsibility  to  reach 
this  contract  award  decision  is  the  Source  Selection  Authority  (SSA).  This  goal  of  the 
decision  process  is  to  provide  the  “best  value”  to  the  Air  Force.  The  best  value  concept 
includes  aspects  such  as  utility,  quality,  cost,  and  risk.  It  frees  the  decision  maker  from 
the  previous  heuristic  of  selecting  the  lowest  cost  proposal  and  allows  him  to  reach  his 
own  decision  based  on  the  breadth  of  data.  A  Source  Selection  Evaluation  Team 
(SSET)  assists  the  SSA  in  gathering  and  consolidating  information. 
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The  decision  scenario  is  a  complex  one  with  several  competing  factors  and 
decision  criteria.  The  decision-maker  (SSA)  is  tasked  with  sorting  through  the  great 
amounts  of  information,  analyzing  it,  and  determining  the  best  value  for  the  Air  Force. 
With  such  a  formidable  task,  decision-makers  can  become  overwhelmed  with  the  task 
and  can,  in  turn,  oversimplify  the  decision  through  the  omission  of  factors  relevant  to  the 
decision.  If  these  decisions  are  made  without  taking  into  account  all  the  relevant 
information  and  comparison  factors,  it  is  very  likely  the  decisions  will  be  less  than 
optimal.  In  other  words,  the  best  value  proposal  may  not  be  selected  for  contract 
award.  The  repercussions  of  not  selecting  the  best  value  proposal  could  be  a  weapon 
system  that  does  not  provide  the  capability  the  warfighters  need  to  accomplish  their 
mission.  Money  could  be  wasted  and  people  could  die  as  a  result  of  a  poor  source 
selection  decision.  With  so  much  on  the  line  in  a  complex  decision  scenario,  all  possible 
aids  should  be  provided  to  the  SSA  to  enable  him  to  effectively  analyze  the  information, 
compare  decision  factors  and  reach  the  best  value  decision.  One  of  these  aids  is  the 
decision  support  system  (DSS). 

DSS  technology  has  been  available  for  a  number  of  years.  There  are  a  number 
of  commercial  providers  of  DSS  software  that  could,  with  minimal  changes,  support  the 
specific  source  selection  decision  process.  The  technology  is  available  and  the  source 
selection  process  could  possibly  benefit  from  the  implementation  of  DSS  technology. 

Specific  Problem 

The  source  selection  process  tasks  the  SSA  with  reaching  an  optimal  decision 
addressing  numerous  decision  factors  and  a  plethora  of  information.  Studies  by  George 
Miller  have  shown  that  humans  can  “distinguish  between  only  about  seven  choices,  plus 
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or  minus  two”  (Miller,  1956:  84).  Given  the  limitations  of  humans  as  information 
processors,  perhaps  this  process  could  be  improved  with  an  information  processing  and 
decision  aid  like  a  DSS. 

The  purpose  of  this  study  is  to  determine  the  impact  of  using  a  DSS  to  arrive  at  a 
source  selection  decision.  Provided  this  quantitative  information,  the  following  question 
must  be  addressed:  given  the  available  technology  and  its  detailed  impact,  could  DSS 
technology  improve  the  source  selection  process?  This  question  includes  not  just  the 
Impact  of  DSS  on  decision  factors  but  must  also  take  into  account  broader  issues  like 
technology  as  a  barrier,  the  corporate  culture,  potential  personal  resistance,  and 
perception  problems.  This  research  question  will  be  broken  down  In  this  study  and 
addressed  as  a  grouping  of  several  research  questions.  They  are  listed  below: 

How  does  DSS  technology  affect  key  decisional  factors  such  as  time, 
confidence,  and  quality? 

Have  the  perceptions  concerning  DSS  technology  changed  within  the  last  five 
years? 

How  will  DSS  technology  affect  the  satisfaction  with  the  source  selection 
decision  process? 

How  will  DSS  technology  affect  the  alternatives  considered  prior  to  reaching  a 
decision? 

Do  individuals  involved  in  the  source  selection  process  foresee  DSS  technology 
being  implemented  in  the  decision  process? 
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The  source  selection  decision  process  is  a  complex  process  demanding  almost 
superhuman  capabilities  from  the  SSA.  The  amount  of  information  that  must  be 
considered  and  the  number  and  complexity  of  the  decision  factors  is  immense.  Since 
DSS  have  shown  to  capable  facilitators  of  complex  decision  processes,  the  natural 
conclusion  would  be  to  implement  DSS  technology  into  the  process.  This  study  will 
attempt  to  determine  the  impact  of  DSS  technology  on  the  source  selection  decision  as 
well  as  ascertain  the  perceptions  of  this  technology  within  the  source  selection 
community. 

This  study  will  address  these  research  questions  by  first  providing  a  background 
of  the  research  areas  relevant  to  this  study  in  a  thorough  review  of  the  available 
literature.  A  description  of  the  hypotheses  to  be  tested  in  this  study  and  how  they  will  be 
tested  is  included  in  Chapter  III,  Methodology.  Chapter  IV  describes  the  results  of  the 
hypothesis  testing  and  the  Conclusion  summarizes  the  impact  of  the  results. 
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II.  Literature  Review 


Introduction 

Decision-making  is  the  choice  between  alternatives  and  is  an  integral  part  of  all 
aspects  of  human  life.  Decision-making  involves  gathering  information,  processing  it, 
comparing  alternatives,  and  choosing  between  the  alternatives  (Barr  and  Sharda,  1997: 
133-134).  It  is  something  we  all  do  regularly. 

A  Decision  Support  System  (DSS)  is  a  system  of  rules  designed  to  aid  in  the 
decision  making  process.  This  discussion  of  DSS  will  focus  on  computer-based  DSS, 
where  the  rules  are  programmed  in  the  software.  The  DSS  does  not  make  the  decision 
for  the  user  but  instead  is  a  tool  used  to  assist  In  the  compilation  and  comparison  of 
alternatives.  The  ultimate  decision  rests  with  the  individual  user  (Barr  and  Sharda,  1997: 
134).  This  thesis  documents  the  examination  of  whether  DSS  usage  could  improve  the 
source  selection  decision  process  and  provides  a  framework  for  further  investigation  into 
the  feasibility  of  using  DSS  to  assist  in  the  source  selection  decision  process. 

The  source  selection  process  is  an  established  governmental  process  that 
chooses  between  different  contractor’s  proposals.  The  process  begins  when  needs  are 
identified  by  the  government  and  Requests  For  Proposal  (RFP)  are  made  available  to 
potential  contractors.  Contractors  reply  to  the  RFP  with  their  complete  technical, 
contractual,  and  cost  solution.  The  proposals  are  then  examined  according  to  the 
“technical,  financial,  and  economic  or  business  considerations  consistent  with  the 
requirement  [RFP]  and  business  and  legal  constraints”  (AFFARS  Appendix  BB,  1997: 

2).  The  governing  regulation  for  source  selection  decisions  also  allows  for  the  possibility 
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of  utilizing  a  computer-based  DSS  when  it  states  “each  member  [of  the  process]  shall  be 
given  access  to  the  full  range  of  evaluation  tools  available"  (AFFARS  Appendix  BB, 

1997:  7). 

DSS  use  has  shown  to  affect  several  aspects  of  the  decision  process  with  other 
decision  types  and  perhaps  these  positive  results  will  also  be  seen  with  DSS  use  in  the 
source  selection  decisions  (Eierman  et  al.,  1995: 8).  While  DSS  use  has  no  significant 
affect  on  the  time  required  to  make  the  decision,  it  has  shown  to  increase  the  number  of 
decision  alternatives  considered,  increase  decision  confidence,  and  increase  decision 
quality  (Eierman  et  al.,  1995:  8-17).  The  research  question  for  this  proposal  is  whether 
these  effects  discovered  in  the  literature  will  also  be  seen  with  DSS  usage  in  the  source 
selection  decision  process.  More  specifically,  the  research  question  is  whether  DSS  use 
in  the  source  selection  process  will  have  no  effect  on  time  required  to  make  a  decision, 
increase  the  number  of  decision  alternatives  considered,  increase  decision  confidence, 
and  increase  decision  quality.  To  address  this  research  question,  we  will  address  how 
decisions  are  made,  what  factors  are  affected  by  DSS  usage,  source  selection 
decisions,  and  hypothesize  a  result  when  DSS  are  used  in  source  selection  decisions. 

The  source  selection  decision  process  could  benefit  from  the  introduction  of  DSS 
to  the  process  (Vickery,  1989:  3).  The  fact  that  DSS  usage  increases  the  number  of 
alternatives  considered  increases  decision  confidence,  and  increases  decision  quality 
proves  important  to  this  study  (Eierman  et  al.,  1995:  8-17).  Since  these  decisional 
factors  are  also  important  to  the  more  specific  source  selection  decision,  this  thesis  will 
address  whether  the  same  relationship  exists  between  DSS  usage  and  improvement  in 
source  selection  alternatives  considered,  confidence,  and  quality. 
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Summary  of  Current  Knowledge 

To  effectively  investigate  this  question,  this  literature  review  will  address  three  main 
areas.  First,  the  factors  affecting  good  decisions  will  be  defined.  Secondly,  DSS  will  be 
defined  and  the  effects  of  DSS  usage  on  decisions  will  be  addressed.  Finally,  the 
literature  review  will  define  the  source  selection  process  and  look  into  the  possibilities  of 
DSS  usage  in  this  decision  scenario. 

Decision-Making 

Decision-making  has  been  understood  as  an  integral  part  of  human  existence  for 
some  time  and  decisions  have  generally  been  classified  into  two  categories, 
programmed  and  nonprogrammed  (Simon,  1977:  45).  Simon  describes  programmed 
decisions  as: 

Decisions  are  programmed  to  the  extent  that  they  are  repetitive  and  routine,  to 
the  extent  that  a  definite  procedure  has  been  worked  out  for  handling  them  so 
they  don’t  have  to  be  treated  de  novo  each  time  they  occur.  (Simon,  1977: 46) 

Simon  describes  nonprogrammed  decisions  as; 

Decisions  are  nonprogrammed  to  the  extent  that  they  are  novel,  unstructured 
and  usually  consequential.  There  is  no  cut-and-dried  method  for  handling  the 
problem  because  it  hasn’t  arisen  before,  or  because  its  precise  nature  and 
structure  are  elusive  or  complex,  or  because  it  is  so  important  that  it  deserves  a 
custom-tailored  treatment.  (Simon,  1977:  46) 

It  is  important  to  note  that  the  research  indicates  these  two  decision  types  are  not 

mutually  exclusive,  but  that  decisions  generally  have  characteristics  of  both.  Simon 

describes  this  by  stating,  “The  distinction  [between  programmed  and  nonprogrammed] 

will  be  a  convenient  one.  I  shall  use  it...  hoping  that  the  reader  will  remind  himself  from 

time  to  time  that  the  world  is  mostly  gray  with  only  a  few  patches  of  black  and  white” 

(Simon,  1977;  47).  This  is  important  since  both  of  these  extremes,  programmed  and 
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nonprogrammed,  could  not  effectively  apply  DSS  technology.  Programmed  decisions 
would  not  require  extensive  decision-making  capabilities  since  a  heuristic  could  be 
developed  and  applied  to  every  decision  since  they  are  all  the  same.  Nonprogrammed 
decisions  would  be  so  novel  that  it  would  be  extremely  difficult  to  develop  a  new  DSS  for 
every  decision.  DSS  technology  applies  to  the  decision  in  this  “gray  area.” 

The  source  selection  decisions  to  be  addressed  in  this  study  can  be  described 
as  between  programmed  and  nonprogrammed,  in  the  “gray  area”  as  Simon  called  it. 
These  decisions  are  programmed  to  the  extent  that  they  are  repetitive  and  recurring. 

For  example,  the  source  selection  decision  process  is  common  within  the  contracting 
arena  and  the  goal  of  every  source  selection  is  the  same,  best  value  (AFFARS 
Appendix  BB,  1997;  3).  In  this  respect,  the  source  selection  decision  is  programmed. 
Source  selection  decisions  are  nonprogrammed  to  the  extent  that  they  produce  a 
consequential  decision  and  there  is  no  one  way  to  solve  all  source  selection  decisions. 
The  source  selection  process  decides  which  companies  the  military  will  do  business 
with.  These  decisions  could  have  far-reaching  consequences  in  our  national  defense. 
These  decisions  are  also  nonprogrammed  because  there  can  be  no  one  formula  for 
computing  the  best  value  of  different  source  selection  proposals.  It  is  different  every 
time.  The  blending  of  programmed  and  nonprogrammed  decision  type  characteristics 
within  the  source  selection  decision  process  lead  it  to  be  classified  as  a  combination 
decision  type. 

DSS  has  been  found  to  add  structure  to  the  decision-making  process.  The 
added  structure  of  a  DSS  could  alter  the  decision  type,  in  effect  pushing  the  decisions 
more  toward  the  programmed  extreme.  Decisions  previously  classified  as 
nonprogrammed  may,  with  the  structure  of  a  DSS,  become  combination  decision  types. 
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The  Rational  Model 


The  rational  model  was  one  of  the  first  models  to  describe  how  decisions  are 
made.  It  is  composed  of  three  phases,  intelligence,  design,  and  choice  (Simon,  1977: 
40).  During  the  intelligence  phase,  data  is  gathered  on  goals,  objectives,  and  alternative 
courses  of  action.  The  design  phase  is  where  the  decision-maker  designs  the  method 
by  which  the  alternatives  will  be  evaluated.  Finally,  the  decision  is  made  in  the  choice 
phase. 

The  rational  model  represents  an  idealistic  approach  which  assumes  that  all  the 
relevant  information  needed  to  make  a  decision  is  available,  and  that  the  decision-maker 
is  rational  and  able  to  interpret  all  of  the  information.  The  result  of  this  model  is  the 
optimal  decision.  In  order  to  reach  the  optimal  decision,  a  completely  rational  decision¬ 
maker  must  base  his  decision  on  perfect  knowledge  of  all  the  alternatives  (Simon,  1976: 
xxviii).  Hogarth  1987  also  describes  the  optimizing  process  as  “choosing  the  best  of  all 
possible  alternatives”  (Hogarth,  1987:65).  This  involves  assessing  every  possible 
alternative  according  to  costs  and  benefits  and  selecting  the  rational  alternative. 

Given  the  human  information-processing  limitations,  optimizing  is  a  difficult  goal 
to  achieve.  Simon  questioned  this  model  in  the  1950’s  when  he  showed  that  human 
information-processing  limitations  preclude  rational  investigation  of  all  possible 
alternatives  (Simon  1955:  69). 

Problems  with  the  Rational  Environment 

There  are  several  reasons  why  the  rational  model  is  not  a  true  representation  of 
how  humans  make  decisions.  They  are  based  on  the  innate  limitations  of  the  human 
mind  that  prevent  decision-makers  from  making  a  rational  choice.  Hogarth  1987  lists 
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four  reasons  for  the  existence  of  the  disparity  between  the  rational  model  and  how 
decisions  are  actually  made.  These  reasons  are;  randomness  and  the  probabilistic 
environment,  information  issues,  limited  resources,  and  bounded  rationality  (Hogarth, 
1987:  204-205).  The  following  section  will  first  discuss  the  innate  human  information¬ 
processing  limitations  and  then  address  Hogarth’s  four  points  individually. 

Information-Processing  Limitations 

Miller  1956  proposed  a  theory  on  the  limitations  of  the  human  mind  called  the 
“magical  number  seven."  Miller  believes  that  the  human  judgement  and  short-term 
memory  limits  the  amount  of  information  we  are  able  to  receive  process  and  remember. 
Studies  conducted  by  Miller  showed  that  people  were  able  to  distinguish  between  only 
about  seven  choices,  plus  or  minus  three  (Miller,  1956:  84).  This  shows  an  innate 
human  information  processing  limitation  that  could  influence  the  decision  making 
process.  Since  decision  quality  has  been  positively  correlated  with  the  number  of 
alternatives  considered  in  reaching  the  decision  (Cats-Baril  and  Huber,  1987:  368), 
humans  could  make  better  quality  decisions  with  the  processing  assistance  of  a 
computer-based  DSS.  The  DSS  enables  the  decision-maker  to  distinguish  between 
more  than  seven  plus  or  minus  two  choices. 

The  main  process  that  appears  in  the  literature  to  describe  how  humans,  with 
their  information  and  processing  limitations,  actually  make  decisions  is  satisficing. 
Satisficing  is  finding  a  decision  that  is  a  “satisfactory  alternative”  and  this  approach  is  a 
more  accepted  decision-making  approach  than  optimizing  (Hogarth,  1987:65).  Since 
every  possible  alternative  cannot  be  assessed  rationally  according  to  the  cost-benefit 
analysis,  the  decision-maker  settles  for  a  decision  that  is  acceptable,  though  it  may  not 
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be  the  optimal  decision.  Simon  states  that  decision-makers  satisfice  by  relying  on  habit 
or  established  routines  for  decision  making  (Simon  1977: 48).  The  theory  of  bounded 
rationality  explains  how  decision-makers  cope  with  the  complexity  of  the  decision 
making  process.  Satisficing  allows  for  the  limited  information-processing  capabilities  of 
the  human  mind. 

Randomness  and  the  Probabilistic  Environment 
Humans  have  difficulty  dealing  with  the  uncertain  environment  we  live  in  and  in 
order  to  cope,  humans  are  apt  to  believe  that  the  environment  is  probabilistic  instead. 
Hogarth  states  that  “the  world  is  perceived  by  us  as  being  probabilistic  since  we  are 
unable  to  see  and  comprehend  the  myriad  factors  that  cause  events  to  occur”  (Hogarth, 
1987: 12).  This  inability  to  understand  the  world  around  us  is  based  on  our  limited 
information-processing  capabilities  and  leads  us  to  seek  out  patterns  and  probabilities  of 
things  we  observe.  The  human  tendency  to  perceive  the  environment  as  being 
probabilistic  goes  against  the  rational  model  because  in  the  rational  model  we  would 
have  perfect  Information  and  understanding  and  not  be  relegated  to  approximating  false 
probabilities. 


Information  Issues 

Difficulty  with  processing  large  amounts  of  information  leads  people  to  take 
shortcuts  in  the  judgement  and  choice  stages  of  the  decision-making  process  prohibiting 
their  attainment  of  the  rational  model.  Examples  of  these  are  the  availability  heuristic 
and  the  anchoring  and  adjusting  heuristic.  The  availability  heuristic  states  that  decision¬ 
makers  are  likely  to  base  their  decisions  on  information  that  is  readily  available  (Tversky 
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and  Khaneman,  1973:  208).  Simply  put,  Tversky  and  Khaneman  state  that,  contrary  to 
the  rational  model,  the  availability  heuristic  “uses  strength  of  association  as  a  basis  for 
the  judgement  of  frequency”  (Tversky  and  Khaneman,  1973:  208).  This  greatly  limits 
the  information  search  and  falls  far  short  of  the  rational  model’s  “perfect  information” 
constraint  (Hogarth,  1987:  53). 

People  make  estimates  by  starting  from  an  initial  value  that  is  adjusted  to  reach 
a  final  answer  (Tversky  and  Khaneman,  1974: 1127).  These  “adjustments  are  typically 
insufficient”  and  “different  starting  points  yield  different  estimates,  which  are  biased 
toward  the  initial  values”  (Tversky  and  Khaneman,  1974: 1127).  The  anchoring  and 
adjusting  heuristic  deals  with  the  propensity  of  decision-makers  to  rely  on  available 
information  (availability  heuristic)  and  arrive  at  a  preliminary  starting  point.  Information 
that  is  gathered  and  processed  after  the  anchor  has  been  set  will  cause  only  minor 
adjustments  to  the  original  anchor  (Hogarth,  1987: 54).  This  heuristic  also  prevents 
humans  from  arriving  at  a  truly  rational  decision.  The  two  heuristics  described  above 
deal  with  the  two  main  points  of  information  issues  as  describes  by  Hogarth,  information 
availability  and  information  processing.  The  availability  heuristic  shows  how  humans 
use  only  information  that  is  easily  available,  precluding  the  attainment  of  perfect 
knowledge  as  the  rational  model  suggests.  The  anchoring  and  adjusting  heuristic 
explains  how  humans,  unable  to  process  all  the  information,  anchor  early  in  the  decision 
process  and  tend  to  adjust  only  slightly  from  there.  This  is  also  contrary  to  the  rational 
model. 
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Limited  Resources 


A  limitation  on  essentially  all  decision  processes  is  limited  resources  (Simon, 
1982:  67).  These  resource  limitations  may  be  time  or  budget  constraints  within  which 
the  decision-maker  must  reach  a  decision.  This  situation  of  limited  resources  makes  it 
extremely  difficult  for  the  decision-maker  to  gather  all  the  relevant  information,  process  it 
completely,  and  make  a  rational  decision  as  purported  by  the  rational  model.  We  all 
face  limited  resources  when  confronted  with  a  decision  process  whether  it  is  time  or 
money  and  this  limitation  makes  the  rational  model  difficult  if  not  impossible  to  follow. 

Bounded  Rationality 

Simon  developed  the  theory  of  bounded  rationality.  He  states  that  the  increasing 
complexity  of  decisions  “has  stimulated  the  development  of  new  kinds  of  models  of 
rational  decision  that  take  special  account  of  the  very  limited  information-gathering  and 
computing  capacity  of  human  beings”  (Simon,  1982:  423).  In  his  book  Judgement  and 
Choice,  Hogarth  also  discusses  the  theory  of  bounded  rationality.  This  states  that 
“rational  choice  requires  having  both  much  information  at  one’s  disposal  and  impressive 
computational  capacity”  (Hogarth  1987:  64).  He  goes  on  to  state  that  no  one  has 
“probably  ever  made  a  choice  of  any  consequence  involving  all  the  elements  of  the 
rational  model”  (Hogarth  1987:  64).  In  order  to  cope  with  the  complicated  nature  of 
decision  problems,  humans  reduce  the  scope  of  the  problem  resulting  in  decisions  that 
can  be  called  reasonable  but  not  rational  (Hogarth,  1987:  65).  Humans  reduce  the 
scope  of  the  problem  by  imposing  constraints  that  are  not  part  of  the  problem  but  make 
the  decision  process  more  understandable.  The  human  mind  is  able  to  neither  gather 
all  the  information  nor  process  it  rationally. 
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The  literature  review  of  decision-making  details  many  important  aspects  of  how 
humans  make  decisions  and  how  this  decision-making  rarely,  if  ever,  follows  the  rational 
model.  The  original  goal  has  been  the  rational  decision.  It  is  important  to  note  that 
decision  quality  has  been  shown  to  improve  to  the  extent  that  humans  are  able  to 
approach  the  rational  model,  gather  all  the  available  information,  process  it,  and 
rationally  select  the  best  alternative  (Simon,  1976:  xxviii).  The  rational  model  is  the 
goal.  Rational  decision-making  has  also  been  shown  to  be  an  impossible  goal  given  the 
satisficing  process.  Miller’s  “magical  number  seven,”  and  Hogarth’s  four  points.  To 
improve  the  decision  process  and  get  closer  to  the  goal  of  rational  decision-making, 
computer’s  information  processing  capabilities  could  moderate  the  above-noted 
concerns.  The  computer  effect  could  supplement  the  human  mind  and  increase  the 
rationality  of  the  decision-making  process.  The  mitigating  effect  of  DSS  on  several  of 
these  points  could  improve  the  rationality  of  the  decision  process.  An  example  is  the 
ability  of  computers  to  process  and  compare  large  amounts  of  information  could  mitigate 
the  effects  of  our  innate  information-processing  limitations.  Another  example  of  this  is 
how  the  decision-making  structure  programmed  into  the  DSS  could  lessen  the  reliance 
upon  the  heuristics  described  by  Simon  and  Hogarth,  thereby  increasing  the  rationality 
of  the  decision  process. 

Decision-Making  Model 

Mintzberg  et  al.  1976  developed  a  model  for  decision-making  based  on  three 
stages:  identification,  development,  and  selection  (Mintzberg  et  al.,  1976: 266).  A 
simplified  model  of  decision-making,  based  loosely  on  Mintzberg’s  model,  is  in  Figure  1. 
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Figure  1 .  Decision  Process  Model 


The  model  depicted  above  shows  the  decision-making  process  as  it  will  be  discussed  in 
this  study.  The  decision  process  begins  with  problem  recognition.  During  this  phase, 
the  realization  is  made  that  a  decision  on  the  subject  is  needed.  The  main  question  that 
needs  to  be  answered  in  this  phase  is  whether  what  is  the  root  problem  to  be  studied. 
Though  this  seems  rather  simple,  it  is  a  critical  and  often  difficult  stage  in  the  decision¬ 
making  process.  In  source  selection  decisions,  this  stage  is  straightforward  since  the 
problem  is  always  known  and  stated,  namely,  finding  the  ‘best  value’  proposal. 

Once  the  problem  is  recognized,  the  information-gathering  phase  begins.  During 
this  phase,  information  on  previous  similar  decisions  is  gathered  as  well  as  specific 
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information  on  this  particular  decision.  Two  important  factors  in  this  stage  In  the  process 
are  whether  enough  information  has  been  gathered  to  approach  rationality  and  whether 
the  information  gathered  provides  adequate  subject  knowledge.  In  the  source  selection 
decision  process,  the  SSET  and  not  the  decision-maker  (SSA)  complete  the  information 
gathering.  The  information-gathering  phase  is  structured  to  the  extent  that  each 
proposal  must  address  the  complete  list  of  topics  presented  in  the  RFP. 

From  this  information,  a  determination  of  the  type  of  decision  can  be  made. 
Source  selection  decisions,  as  addressed  above,  fall  into  the  combination  category. 

This  category  contains  aspects  from  both  programmed  and  nonprogrammed  decision 
types  and  is  able  to  be  supplemented  by  DSS  technology. 

The  final  step  is  the  decision/choice  stage.  Once  the  background  information  is 
gathered,  a  decision  must  be  made.  Factors  of  the  choice  phase,  and  listed  in  the 
decision/choice  box  above,  include  time,  confidence,  quality,  number  of  decision 
analyses  considered,  and  decision  satisfaction.  The  time  required  to  make  a  decision  is 
important  because  of  the  limited  resources  we  all  operate  under.  Confidence  and 
satisfaction  with  the  decision  are  important  factors  to  the  decision-maker  and  tell  about 
the  decision-maker’s  perceptions  of  the  decision  process.  Decision  quality  is  the 
ultimate  goal.  It  can  be  described  as  the  degree  of  rationality.  The  best  decision  will  be 
the  rational  one  and  decision  quality  increases  to  the  extent  that  we  approach  rationality. 
The  more  decision  analyses  addressed  in  the  decision-making  process  the  closer  we 
get  to  approaching  the  rational  model’s  goal  of  looking  at  ail  alternatives.  All  of  these  are 
factors  of  the  decision  and  will  be  used  to  assess  the  decision  itself.  This  model  will  be 
used  throughout  this  study  to  visually  describe  the  decision  process. 
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Decision  Support  Systems 

Computers  do  not  have  the  same  restrictions  and  limitations  as  humans  with 
respect  to  the  amount  of  information  they  are  able  to  process.  Thus,  they  may  help  the 
decision-maker  efficiently  process  the  large  amounts  of  information,  compare 
alternatives,  and  therefore  reach  an  optimized  decision.  “Advances  in  information 
retrieval,  processing,  and  display  technologies  have  certainly  led  to  computer  programs 
that  help  people  perform  management  functions.  Since  the  purpose  of  these  systems  is 
to  support  managers  responsible  for  making  and  implementing  decisions  rather  than 
replace  them,  these  applications  are  called  Decision  Support  Systems  (DSS)”  (Alter, 
1980: 1).  Barr  and  Sharda  1997  describe  DSS  by  stating  “DSS’s  major  value  is  in 
removing  information  overload  and  redundancy  by  summarizing,  categorizing,  and 
projecting  important  data,  thus  decreasing  the  cognitive  effort  required  to  process  large 
amounts  of  information  and  allowing  decision-makers  to  focus  on  more  central  elements 
and  issues  in  the  decision  process”  (Barr  and  Sharda,  1997: 134). 

The  DSS  supports  the  processing  of  information  and  comparison  of  alternatives 
and  leaves  the  actual  decision  up  to  the  decision-maker.  DSS  are  suited  for 
programmed  decisions  and  decisions  in  the  “gray  area,”  as  Simon  called  it,  between 
programmed  and  nonprogrammed  decision  types.  DSS  are  regularly  used  in 
programmed  decisions  since  these  problems  are  routine  and  repetitive  and  a  DSS  can 
be  developed  to  automate  the  decision  process.  An  example  of  this  is  a  credit  approval 
system.  Pertinent  information  is  entered  into  the  DSS  (income,  debt,  etc.)  and  the  DSS 
determines  the  credit  line  by  comparing  several  alternatives.  Nonprogrammed  decisions 
are  so  novel  that  they  defy  DSS  development  (Keen,  1980: 27).  It  is  generally 
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considered  not  worth  the  time  required  to  construct  a  DSS  for  one  decision  that  will  not 
be  repeated.  Semi-structured  tasks  allow  for  a  combination  of  human  judgment  and 
computer  capabilities  (Keen,  1980:  27). 

DSS  can  be  added  to  the  decision  process  model  discussed  above.  Once  the 
decision  type  is  assessed,  a  determination  of  the  benefits  of  adding  a  DSS  can  be 
made.  If  a  DSS  is  used  in  the  process,  it  could  impact  several  factors  in  the 
decision/choice  stage.  The  introduction  of  DSS  is  depicted  in  the  model  below. 
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Figure  2.  Decision  Process  Model  with  DSS  Effects 
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The  question  is  how  the  introduction  of  a  DSS  into  the  decision  process  influences  the 
decision.  The  impact  of  using  a  DSS  on  certain  factors  in  the  model  will  be  addressed  in 
the  following  section. 

DSS  Effect  on  Decision  Factors 

DSS  effectiveness  has  been  evaluated  in  many  studies.  The  research  indicates 
that  DSS  usage  affects  several  factors  in  the  decision  process.  The  following  literature 
on  DSS  shows  the  relationships  between  DSS  usage  and  several  independent 
variables.  The  variables  addressed  include  time  required  to  make  a  decision,  the 
number  of  alternatives  addressed,  confidence  in  the  decision,  satisfaction  with  decision, 
and  decision  quality. 

Time  to  Make  a  Decision 

DSS  use  is  new  to  many  of  the  subjects  tested  in  the  literature  and  therefore,  the 
learning  curve  effect  is  seen  in  the  majority  of  studies  done  over  time.  Barr  and  Sharda 
1997  stated  that  DSS  use  leads  to  “decreased  time  spent  in  processing  information...  by 
letting  decision-makers  focus  on  strategic  issues”  (Barr  and  Sharda,  1997: 134).  In 
other  words,  the  DSS  assists  in  simplifying  and  speeding  the  decision  process  by 
decreasing  the  “cognitive  effort  required  to  process  [the]  large  amounts  of  information” 
(Barr  and  Sharda,  1997: 134). 

Another  study  by  Sharda  et  al.  discovered  that  in  the  first  three  periods  of  use, 
the  time  required  to  make  a  decision  for  those  using  DSS  was  significantly  greater  than 
for  those  who  did  not  use  a  DSS.  From  the  fourth  period  of  use  to  the  end  of  the  study, 
the  differences  in  time  required  between  the  two  groups  became  insignificant  (Sharda  et 
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al.,  1988:  153).  Mackay  et  al.’s  study  revealed  similar  results.  It  found  that  novice  DSS 
users  took  significantly  longer  than  decision-makers  not  using  the  DSS.  This  study  also 
found  that  experienced  DSS  users  required  significantly  less  time  to  make  a  decision 
than  the  novice  DSS  users  (Mackay  et  al.,  1992:  660-663).  The  time  factor  seems  to 
follow  the  learning  curve,  whereby  originally  decisions  with  DSS  take  longer  than  those 
without  but  this  difference  disappears  after  the  subjects  have  become  accustomed  to 
the  new  technique. 

Confidence  in  Decision 

Most  evidence  in  the  literature  concerning  decision  confidence  shows  that  use  of 
a  DSS  has  a  slight  positive  effect.  While  a  study  by  Cats-Baril  and  Huber  showed  no 
significant  difference  in  decision  confidence  between  DSS  users  and  non-users  (Cats- 
Baril  and  Huber,  1987:  370),  Sharda  et  al.  found  a  slight  increase  in  confidence  with 
DSS  usage  (Sharda  et  al.,  1988: 154).  A  study  by  Schroeder  and  Benbasat  found  that 
since  DSS  users  processed  significantly  more  information  that  non-DSS  users  and 
believed  a  positive  relationship  existed  between  the  amount  of  information  and  the 
confidence  in  the  decision,  they  would  have  greater  confidence  in  the  DSS  decisions 
(Schroeder  and  Benbasat,  1975:  564).  A  study  by  Barr  and  Sharda  1997  stated  that 
DSS’s  ability  to  summarize,  categorize,  and  project  data  decreased  the  cognitive  effort 
required  and  “increased  the  confidence  in  the  decision”  (Barr  and  Sharda,  1997:134). 
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Decision  Quality 


Decision  quality  is  defined  as  the  ability  to  reach  the  optimal  decision.  Increasing 
decision  quality  is  the  most  significant  impetus  for  using  a  DSS.  The  results  on  DSS’s 
influence  on  decision  quality  vary  a  great  deal.  Sharda  et  al.’s  study  noted  significantly 
greater  decision-making  performance  by  people  using  a  DSS  when  compared  with 
those  who  did  not  (Sharda  et  al.,  1988:155).  This  study  was  composed  of  eight 
separate  periods  of  DSS  use  with  measurements  of  decision  quality  taken  for  each 
period.  It  is  interesting  to  note  that  the  significant  differences  in  decision  quality 
between  the  DSS  group  and  the  non-DSS  group  was  not  realized  until  the  fourth  period. 
This  gives  credence  to  the  learning  curve  effect.  From  the  fourth  period  on,  the 
differences  were  significant,  as  was  the  overall  difference.  Cats-Baril  and  Huber’s  study 
on  DSS  showed  no  significant  difference  in  decision  quality  between  DSS  groups  and 
non-DSS  groups  (Cats-Baril  and  Huber  1987:  363).  A  possible  explanation  for  this  is 
the  fact  that  this  study  measured  only  one  period  and  did  not  allow  for  the  users  to 
progress  along  the  learning  curve. 

Eierman  et  al.’s  compilation  of  DSS  research  discovered  a  positive  impact  on 
decision  quality.  He  states,  “results  from  those  studies  comparing  a  ‘DSS’  treatment 
with  a  ‘no  DSS’  alternative  show  that...  DSS  can  have  a  positive  impact  on  performance 
[decision  quality]”  (Eierman  et  al.,  1995: 10-11).  Barr  and  Sharda  1997  also  supported 
these  results.  They  stated  that  “numerous  studies  have  documented  improvements  in 
quality  of  decision  outcomes  through  introduction  and  use  of  DSS”  (Barr  and  Sharda, 
1997: 133).  The  continue  by  stating  that  “laboratory  studies  employing  ‘objective’ 
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measures  of  decision  performance  and  using  repeated  measures  did  indicate  that  DSS 
availability  could  be  expected  to  increase  decision  quality”  (Barr  and  Sharda,  1997: 
135). 


Decision  Analyses  Considered 

It  is  consistently  noted  in  the  literature  that  decision-makers  using  DSS  address 
more  alternatives  than  decision-makers  not  using  the  system.  Cats-Baril  and  Huber 
noted  in  their  study  that  DSS  “enhanced  productivity  more  (generated  higher  number  of 
objectives  and  alternatives)  than  passive  aids.  Since  high  productivity  of  ideas  has  been 
correlated  with  larger  numbers  of  high-quality  ideas,  the...  interactive  presentation 
[DSS]  appears  to  provide  the  most  effective  support  in  terms  of  achieving  decision 
quality”  (Cats-Baril  and  Huber,  1987:  368).  A  study  by  Sharda  et  al.  also  found  that 
users  of  DSS  investigated  more  alternatives  than  those  who  did  not  use  the  DSS 
(Sharda  et  al.,  1988:  154).  A  third  study  by  Mackay  et  al.  supported  the  previous 
studies  by  stating  that  users  of  DSS  “pursued  a  greater  number  of  activities  in  these 
steps  [problem  finding  and  problem  solution],  reflecting  the  ability  of  a  decision-aided 
problem  solver  to  examine  more  efficiently  a  larger  set  of  alternatives”  (Mackay  et  al. 
1992:  665). 

Barr  and  Sharda’s  1997  study  supports  the  increase  in  decision  alternatives 
considered  with  DSS  use.  They  state  “the  use  of  ‘what-if  or  Monte  Carlo  analyses, 
common  in  many  DSS,  allows  the  decision-maker  to  evaluate  numerous  courses  of 
action  and  to  develop  an  increased  understanding  of  how  decision  factors  affect  other 
input  and  outcome  variables”  (Barr  and  Sharda,  1997: 134).  They  also  conclude  that  by 
removing  information  overload,  DSS  use  “lead[s]  to  such  benefits  as  examination  of 
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more  alternatives”  (Barr  and  Sharda,  1997: 134).  Wierenga  and  van  Bruggen  1998  also 
support  the  previous  literature.  They  state  that  the  use  of  DBS  will  “result  in  greater 
number  of  ideas”  (Wierenga  and  van  Bruggen,  1 998:84).  Since  the  rational  model 
centers  on  a  comparison  of  all  possible  alternatives  and  the  goal  is  to  approach 
rationality  in  decision-making,  the  more  alternatives  addressed  in  the  decision  process 
the  closer  to  the  rational  model. 


Satisfaction  with  Decision 

Another  important  aspect  of  a  decision  is  the  satisfaction  with  the  decision 
process.  The  literature  found  a  negative  effect  on  satisfaction  with  DSS  usage. 
Decision-makers  who  used  DSS  “were  less  satisfied  with  the  overall  process”  than 
decision-makers  who  did  not  use  DSS  (Cats-Baril  and  Huber,  1987:  364).  The  Cats- 
Baril  and  Huber  study  found  that  subjects  using  the  DSS  reacted  negatively  to  the 
rigidity  forced  by  the  DSS.  This  negative  reaction  caused  the  low  satisfaction  with  the 
decision  process. 


Table  1.  Effects  of  DSS  Use  on  Key  Decisional  Factors 


Decision  Factor 

Effect  With  DSS  Use  | 

Time  to  make  a  decision 

No  significant  difference  over  time  I 

Confidence  in  decision 

Increases  I 

Decision  quality 

Increases  I 

Number  of  decision  analyses  considered 

Increases  I 

Satisfaction  with  decision 

1  Decreases  | 
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In  summary,  the  literature  indicates  that  DSS  usage  affects  several  decisional 
factors.  Decisions  with  DSS  support  take  more  time  than  without  DSS  support  only  until 
the  users  have  become  accustomed  to  the  software.  The  time  difference  between  the 
two  groups  approached  zero  after  only  three  uses  of  the  DSS  (Sharda  et  al.,  1988: 153). 
Research  also  indicated  that  DSS  usage  increases  the  number  of  alternatives 
considered,  improves  confidence  in  the  decision,  and  improves  decision  quality.  These 
results  seem  to  forecast  an  increased  rationality  with  DSS  use. 

The  Source  Selection  Process 

Within  the  last  five  years,  the  government  has  increased  its  dependence  upon 
contractors  to  provide  many  products  and  services  previously  handled  by  the  active  duty 
military  force.  The  means  by  which  the  government  selects  which  contractor  will  provide 
the  product  or  service  is  the  source  selection  process.  This  discussion  of  the  source 
selection  process  focuses  on  contracts  governed  by  the  Air  Force  Federal  Acquisition 
Regulation  (AFFARS),  Appendix  BB.  This  regulation  deals  with  information  technology 
contracts  of  less  than  $120M  and  other  contracts  of  greater  than  $5M  but  less  than 
$500M  (AFFARS  Appendix  AA,  1997:  2).  Contracts  estimated  to  be  over  these  values 
are  governed  by  AFFARS  Appendix  AA  which  details  a  more  formal  structured  source 
selection  procedure.  Appendix  BB  was  chosen  for  this  study  because  it  is  applicable  to 
the  majority  of  source  selection  processes. 

When  the  government  decides  to  contract  a  service  or  product  out,  the 
government  completes  a  RFP,  requesting  proposals  from  contractors  who  can  provide 
that  product  or  service.  These  proposals  are  examined  by  groups  of  contracting  officers 
and  area  experts,  organized  together  as  the  Source  Selection  Evaluation  Team  (SSET). 
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The  SSET  reviews  and  evaluates  the  proposals  and  ultimately  provides  comparisons  on 
their  individual  area  of  expertise.  The  technical  and  contractual  areas  depicted  above 
would  be  further  divided  according  to  areas  of  expertise.  Below  is  a  chart  depicting  the 
organization  created  during  the  source  selection  process. 


Figure  3.  Source  Selection  Organization 
(James,  1997:  n.pag) 

It  is  important  to  understand  that  the  temporary  organization  depicted  above  is  formed 
for  each  source  selection  (AFFARS  Appendix  BB,  1997:  7).  When  proposals  are 
received  from  prospective  contractors,  they  are  received  by  the  SSET.  The  separate 
technical  and  contracting  teams,  under  supervision  of  the  SSET,  evaluate  the  proposals. 
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The  SSA  does  not  take  part  in  any  of  the  evaluation  at  this  point.  The  SSA  receives  the 
completed  evaluations  and  is  tasked  with  making  a  decision,  having  only  the  information 
compiled  by  the  SSET  at  his  disposal. 

The  goal  of  the  source  selection  process  is  to  select  the  proposal  providing  the 
“best  value”  to  the  government.  The  best  value  criteria  allows  the  government  to  select 
proposals  that  do  more  than  meet  the  minimum  requirements  at  the  lowest  cost. 
Appendix  BB  of  the  Air  Force  Federal  Acquisition  Regulations  (AFFARS)  provides 
guidance  for  the  source  selection  process.  It  mandates  that  the  selection  of  the 
contractor  who  provides  the  best  value.  Best  value  is  described  as  “the  most 
advantageous  offer,  price  and  other  factors  considered,  providing  the  best  mix  of  utility, 
technical  quality,  business  aspects,  risks,  and  price  for  a  given  application”  (AFFARS 
Appendix  BB,  1997:3). 

The  Source  Selection  Authority  (SSA)  is  the  single  most  important  person  in  the 
source  selection  process.  The  SSA  is  an  individual  high-ranking  contracting  officer  who 
has  the  responsibility  of  deciding  which  proposal  will  receive  the  contract.  The  AFFARS 
states  that  “the  SSA  is  responsible  for  the  proper  and  efficient  conduct  of  the  entire 
source  selection  process  encompassing  proposal  solicitation,  evaluation,  selection,  and 
contract  award”  (AFFARS  Appendix  BB,  1997:  7).  The  SSA  is  under  no  obligation  to 
follow  the  advice  provided  by  the  SSET,  but  may  look  at  all  the  information  concerning 
the  proposals  and  make  his/her  own  decision.  The  final  determination  of  which 
contractor  is  selected  for  an  individual  contract  depends  on  one  person’s  decision,  the 
SSA.  “The  SSA  has,  subject  to  law  and  applicable  regulations,  full  responsibility  and 
authority  to  select  source(s)  for  award  and  approve  the  award  of  the  contract(s)” 
(AFFARS  Appendix  BB,  1997:  7).  The  SSA  has  the  tremendous  responsibility  of  sorting 
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through  great  amounts  of  information  and  reaching  the  optimal  decision  for  the 
government.  It  has  been  proposed  that  the  source  selection  process  could  be  improved 
with  the  implementation  of  a  DSS  for  use  by  the  SSA  (Vickery,  1989;  3).  DSS 
technology  assists  in  the  sorting,  displaying,  and  comparing  of  large  amounts  of 
information  such  as  are  involved  in  the  source  selection  process. 

The  AFFARS  Appendix  BB  describes  several  important  aspects  of  the  source 
selection  decision  process.  The  decisional  factors  to  be  discussed  in  this  section  with 
regard  to  the  source  selection  process  are  time  to  make  a  decision,  number  of 
alternatives  considered,  confidence  in  decision,  satisfaction  with  decision,  and  decision 
quality. 


Time  to  Make  a  Decision 

One  important  aspect  of  source  selection  decisions  is  the  time  required  to  make 
a  decision.  The  AFFARS  states  that  the  process  should  be  accomplished  with 
“maximum  efficiency”  (AFFARS  Appendix  BB,  1997;  2).  Kathleen  James,  a 
representative  from  AFMC/PKPA,  detailed  in  her  briefing  slides  to  Air  Force  SSA’s  that 
the  SSP  took  about  75  days  (from  proposal  to  award  decision)  in  1990.  She  stresses  in 
her  slides  that  SSA’s  should  try  to  minimize  the  time  required  to  reach  a  decision 
(James,  1997;  n.pag). 

Confidence  in  Decision 

The  third  important  aspect  of  source  selection  decisions  is  that  SSA’s  must  have 
confidence  in  their  decision.  “You  [SSA’s]  must  be  comfortable  with  your  decision,  and 
you  must  document  its  basis”  (James,  1997;  n.pag).  Since  contractors  can  question 
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source  selection  decision,  SSA’s  must  be  able  to  justify  their  selection  of  the  particular 
proposal  (contractor).  SSA’s  must  be  confident  in  their  decision  in  order  to  effectively 
defend  their  decision  in  the  Source  Selection  Decision  Document  and  subsequent 
briefings  with  contractors  not  selected. 

Decision  Quality 

The  most  important  aspect  of  the  source  selection  decision  is  decision  quality. 
The  AFFARS  states  that  the  “objective  of  the  SSP  is  to  select  the  source  whose 
proposal  has  the  highest  degree  of  credibility  and  whose  performance  can  be  expected 
to  best  meet  the  Government’s  requirements  at  an  affordable  cost”  (AFFARS  Appendix 
BB,  1997:  2).  The  degree  to  which  the  decision  selects  the  ‘best  value’  proposal  is  the 
decision  quality. 

Decision  Analyses  Considered 

A  second  important  aspect  of  source  selection  decisions  is  the  number  of 
analyses  considered.  The  AFFARS  states,  “the  SSA  shall  be  presented  with  sufficient 
in-depth  information  on  each  of  the  competing  offerors  and  their  proposals  to  permit  a 
reasoned,  rational  selection  decision”  (AFFARS  Appendix  BB,  1997:  4).  DSS  increases 
the  amount  of  information  that  can  be  gathered  and  processed,  an  important  part  of  the 
rational  model.  Though  rational  decision-making  has  been  shown  to  be  impossible 
given  the  human  limitations,  perhaps  use  of  a  DSS  can  increase  the  gathering, 
processing,  and  understanding  of  greater  amounts  of  information  and  thus  improve  the 
rationality  of  the  source  selection  decision-making  process. 
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Satisfaction  with  Decision 


The  SSA  training  slides  lists  four  objectives  for  SSA’s  to  keep  in  mind  throughout 
the  process.  One  of  these  objectives  is  user  satisfaction  (James,  1997:  n.pag).  It  is 
important  for  the  SSA  to  be  satisfied  with  the  decision  and  the  process  leading  up  to  the 
decision.  The  briefing  goes  so  far  as  to  say  that  success  entails  SSA  satisfaction  with 
the  decision  (James,  1997:  n.pag). 

In  summary,  there  are  many  factors  important  to  source  selection  decisions. 
They  are  timeliness,  amount  of  alternatives  considered,  confidence,  and  decision 
quality.  The  maximization  of  these  four  factors  will  improve  the  source  selection 
decision  process.  Any  changes  in  the  SSP  should  improve  these  factors. 

Problem 

DSS  technology  has  been  commercially  available  for  over  fifteen  years  but  has 
made  few  inroads  into  the  government  source  selection  process.  This  study  will  look  at 
the  effects  of  DSS  usage  on  source  selection  decisions  and  attempt  to  answer  the 
following  question.  Will  the  introduction  of  DSS  technology  into  the  source  selection 
process  improve  the  time  needed  to  make  a  decision,  the  number  of  decision 
alternatives  considered,  decision  confidence,  and  decision  quality? 

Assumptions 

The  major  assumption  for  this  study  is  that  there  is  a  commercially  available  DSS 
able  to  facilitate  the  source  selection  decision  process.  The  existence  of  this  DSS  will 
be  central  to  the  interview  process.  In  the  interviews,  the  feasibility  of  utilizing  a 


30 


functional  and  available  DSS  will  be  assumed.  This  will  ensure  that  the  interviewees  will 
not  take  into  account  the  development  of  a  new  system,  and  all  the  problems  therein, 
into  their  responses. 

Research  Hypotheses  and  Propositions 

The  methodology  for  this  study  is  two-fold.  First,  experimental  data  was 
analyzed.  This  study  addressed  DSS  usage  in  a  source  selection  decision  setting. 
Empirical  results  from  this  study  should  allow  testing  of  the  following  hypotheses: 

HI :  The  introduction  of  DSS  technology  into  the  source  selection  process 
decreases  decision  time. 

This  hypothesis  addresses  the  impact  of  using  DSS  technology  on  the  time  required  for 
each  person  to  reach  their  decision.  DSS  technology  could  be  ignored  if  it  is  shown  to 
delay  the  source  selection  process. 

H2:  The  introduction  of  DSS  technology  Into  the  source  selection  process 
increases  decision  confidence 

Confidence  addresses  the  belief,  after  reaching  a  decision,  that  they  made  the  right 
decision.  The  confidence  that  the  decision-makers  feel  after  reaching  a  decision  is  an 
important  construct  because  the  decision-makers  must  be  able  to  trust  their  DSS-aided 
decision. 

H3:  The  introduction  of  DSS  technology  into  the  source  selection  process 
increases  decision  quality. 

This  hypothesis  addresses  the  impact  of  using  a  DSS  to  reach  a  decision  on  the 
decision’s  quality.  The  resultant  decision  is  the  most  important  part  of  the  decision 
process  and  the  effect  of  DSS  on  it  is  of  prime  importance. 
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Once  these  hypotheses  are  answered,  we  will  have  information  on  what  impact 
DSS  usage  has  on  the  source  selection  process.  The  second  step  in  this  study  will  be 
to  interview  individuals  involved  in  the  source  selection  process.  During  the  interview, 
the  empirical  results  will  be  discussed  as  a  means  of  determining  their  perceptions  on 
the  feasibility  of  DSS  integration  into  the  source  selection  decision-making  process. 
Examination  of  the  qualitative  interview  data  allows  the  addressing  of  the  following 
propositions; 

P1 :  The  introduction  of  DSS  technology  into  the  source  selection  process 
increases  the  number  of  decision  analyses  considered  prior  to 
reaching  a  decision. 

The  literature  has  shown  that  the  number  of  decision  analyses  conducted  is  positively 
correlated  to  the  quality  of  the  ultimate  decision.  This  proposition  compares  the  number 
of  decision  analyses  considered  in  the  decision-making  process  with  and  without  DSS 
use. 

P2:  The  introduction  of  DSS  technology  into  the  source  selection  process 
increases  satisfaction  with  the  decision  process. 

This  proposition  addresses  the  satisfaction  the  decision-makers  feel  with  the  decision 
process  and  whether  the  satisfaction  will  differ  between  individuals  using  the  DSS  and 
those  who  do  not. 

P3:  The  perception  of  DSS  technology  within  the  source  selection 
community  has  improved  within  the  last  five  years. 
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Computer-based  DSS  technologies  have  met  with  resistance  in  the  past  and  now  that 
computers  are  an  integral  part  of  the  work  environment,  an  improved  perception  of  DSS 
technology  was  proposed.  An  improving  perception  of  the  technology  within  the  source 
selection  community  is  important  if  the  technology  is  to  be  implemented. 

P4:  The  source  selection  community  foresees  DSS  technology  being 
implemented  within  the  decision  process. 

It  is  important  to  determine  whether  the  source  selection  community  believes  this 
technology  has  a  future  in  the  process.  This  proposition  addresses  whether  individuals 
involved  in  the  process  are  of  the  opinion  that  this  technology  will  be  a  part  of  the 
process  in  the  future. 

The  interviewees,  though  not  necessarily  a  representative  sample,  will  provide 
insight  into  propositions  four  through  seven.  Their  active  involvement  within  the  source 
selection  process  will  allow  the  discussion  of  these  propositions  and  the  formulation  of 
questions  for  further  study 

Conclusion 

Decision-making  is  an  integral  part  of  human  existence.  Given  the  human 
information  processing  limitations  discussed  above  by  Simon  and  Miller,  a  computer- 
based  DSS  could  improve  the  decision  process  by  handling  the  information  processing 
and  comparing  tasks  vital  to  decision-making.  The  literature  has  shown  evidence  that 
DSS  usage  improves  the  timeliness,  number  of  alternatives  considered,  confidence,  and 
quality  when  compared  to  decisions  without  DSS  support.  It  has  also  been  shown  that 
source  selection  decisions  also  require  timeliness,  a  maximum  number  of  alternatives 
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considered,  confidence  in  the  decision,  and  decision  quality.  Since  DSS  have  been 
shown  to  increase  the  same  decisional  factors  that  are  important  to  the  source  selection 
process,  I  will  attempt  to  show  that  DSS  will  have  the  same  effects  on  source  selection 
decisions. 

The  following  chapter  will  describe  the  methodology  to  be  used  to  address  these 
hypotheses.  As  stated  above,  the  methodology  will  include  both  empirical  data  from  an 
experiment  and  interviews  conducted  with  experts  in  the  source  selection  community. 
The  methodology  chapter  will  describe  how  the  data  was  gathered,  how  the  hypotheses 
and  propositions  will  be  tested,  and  why  these  methods  were  chosen. 
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III.  Methodology 


Introduction 

This  study  will  use  a  two-pronged  methodology,  consisting  of  the  analysis  of  data 
from  a  previous  experiment  and  an  interview.  The  first  segment  is  an  experiment  on  the 
effects  of  DSS  usage  on  several  decision  variables  to  address  hypotheses  one  through 
three.  The  topics  of  research  design,  discussion  of  variables,  and  discussion  of 
hypotheses  will  be  addressed  in  this  section.  The  second  section  is  a  series  of 
interviews  conducted  with  managers  within  the  source  selection  process  that  will 
address  propositions  one  through  four.  The  interview  questionnaire  will  be  described  in 
this  section. 

Experiment 

The  experiment  and  resultant  data  described  in  this  study  was  accomplished  for 
a  previous  AFIT  thesis  (Vickery,  1989).  This  experiment  effectively  measured  the 
results  of  DSS  usage  on  decision  factors.  At  the  time  this  study  was  conducted,  Likert 
data  could  only  be  manipulated  with  limited  statistical  tests  but  since  that  time  it  has 
been  shown  that  Likert  data  can  be  manipulated  as  interval  data,  allowing  for  increased 
statistical  power.  This  fact  leads  us  to  a  new  examination  of  the  data  taking  advantage 
of  improved  statistical  tests. 
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Research  Design 


The  experiment  conducted  in  this  study  is  a  post-test-only  control  group  design. 
This  design  was  used  to  test  the  three  hypotheses  in  this  study.  The  post-test-only 
control  group  design  uses  one  experimental  group  and  one  control  group  as  depicted 
below  where  Xi  is  the  experimental  treatment  and  O,  and  Ojare  post-test  observations 
(Cooper  and  Emory,  1995:  364). 


(R)  X,  On 

(R)  O2 

The  more  common  research  design  would  have  been  the  pre-test-post-test 
control  group  experimental  design  depicted  below  where  O,  and  O2  denote  the  pre-test 
observations,  X,  denotes  the  experimental  treatment,  and  O3  and  O4  denote  post-test 
observations  (Cooperand  Emory,  1995:  364). 

On  Xi  O3 

O2  O4 

Campbell  and  Stanley  describe  the  pre-test  design  as  a  means  to  ensuring  the 
equality  between  the  experimental  and  control  groups  (Campbell  and  Stanley,  1966: 
25).  The  pre-test  research  design  has  also  been  shown  to  possibly  introduce  bias  into 
the  control  group  that  could  invalidate  the  experimental  results  (Campbell  and  Stanley, 
1966:  14). 

Campbell  and  Stanley  state  that  the  pre-test  notion  is  “not  actually  essential  to 
true  experimental  designs”  (Campbell  and  Stanley,  1966:  25).  They  continue  to  defend 
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the  validity  of  the  post-test  only  design  used  in  this  study  by  stating  “the  most  adequate 
all-purpose  assurance  of  lack  of  initial  biases  between  groups  is  randomization.  Within 
the  limits  of  confidence  stated  by  the  tests  of  significance,  randomization  can  suffice 
without  the  pretest”  (Campbell  and  Stanley,  1966:  25). 

The  experimental  treatment  in  this  study  (Xi)  is  the  use  of  a  decision  support 
system  called  Expert  Choice.  This  decision  software  includes  the  essential  elements  of 
a  DSS.  It  is  based  on  the  Analytical  Hierarchy  Process  that  breaks  down  complex 
decisions  into  competing  component  parts.  The  decision-maker  then  assigns  numerical 
values  to  each  component  and  establishes  the  relationship  between  components.  The 
alternative  comparison  entered  by  the  decision-maker  can  be  done  objectively  or 
subjectively.  The  subjective  comparison  would  merely  state  preference  of  one 
alternative  to  another  while  the  objective  comparison  would  assign  specific  numerical 
values  to  the  relationship  between  alternatives.  The  software  analyzes  the  subsequent 
relationships  and  “graphically  portrays  the  problem,  conflicting  criteria  and  alternatives” 
(Vickery,  1989:  22).  The  Expert  Choice  software  “synthesizes  the  information  and 
provides  a  rank-ordered  list  of  alternatives”  from  the  user-inputted  information  (Vickery, 
1989:  22).  In  this  study,  the  control  group  was  tasked  with  reaching  a  decision  without 
the  Expert  Choice  software  while  the  experimental  group  was  provided  the  software. 
Assignment  to  the  control  and  experimental  groups  was  done  randomly  to  control  for  the 
variables.  This  is  an  essential  part  to  the  post-test-only  design  (Campbell  and  Stanley, 
1966:  25). 
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Procedure 


To  begin  the  experiment,  both  the  experimental  and  control  groups  were  briefed 
on  the  decision  scenario.  The  subjects  of  this  experiment  were  AFIT  Masters 
candidates.  They  were  separated  into  control  and  experimental  groups  randomly.  This 
briefing  included  a  description  of  the  multiple  competing  criteria  as  well  as  the  choices 
available  for  the  ultimate  decision.  The  experimental  group  was  then  introduced  to  the 
DSS  treatment  and  provided  an  explanation  of  the  Analytical  Hierarchy  Process  (AHP). 
Each  individual  within  the  experimental  group  was  provided  an  identical  computer 
loaded  with  the  Expert  Choice  software.  The  control  group  was  asked  to  use  whatever 
decision-making  method  they  felt  most  comfortable  with.  The  post-test  consisted  of  a 
complex  decision  scenario.  After  completing  the  scenario  and  reaching  a  decision,  each 
subject  filled  out  a  questionnaire  designed  to  test  the  variables  to  be  described  in  the 
“Discussion  of  Variables”  section. 

The  post-test  only  experimental  design  effectively  controls  the  threats  to  internal 
validity  (Emory,  1980: 121).  The  major  threat  to  internal  validity  is  whether  the  results 
were  due  to  the  AHP  training  or  the  DSS  itself.  Knowledge  of  the  AHP  is  integral  for  the 
effective  use  of  the  DSS  system.  In  fact,  the  DSS  system  is  based  largely  on  the  AHP. 
Separating  these  two  effects  is  not  necessary  because  the  AHP  is  part  of  the  DSS.  If 
the  results  are  positive,  later  studies  can  investigate  the  individual  roles  of  the  AHP  and 
the  DSS. 

The  post-test-only  control  group  experimental  design  is  not  as  effective  in 
controlling  for  external  validity.  The  two  main  concerns  in  this  area  are  whether  the  test 
affected  the  subjects  and  whether  the  results  can  be  generalized  to  the  population  in 
question.  The  problem  of  the  subjects  being  affected  by  the  act  of  being  tested  is 
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primarily  a  problem  in  the  pre-test-post-test  experimental  design  where  the  pretest 
introduces  unusual  activities  (Emory,  1980: 121).  This  study,  without  a  pre-test,  greatly 
reduces  the  risk  of  this  type  of  threat  to  external  validity.  The  subjects  in  this  study  were 
AFIT  students  who  were  familiar  with  the  acquisition  process,  thereby  also  reducing  the 
risk  of  this  type  of  threat  to  external  validity. 

The  ability  to  generalize  the  results  of  this  study  to  the  population  in  question, 
namely  the  source  selection  authorities  in  the  contracting  arena,  represents  another 
threat  to  external  validity.  The  sample  population  in  our  experiment  was  limited  to  forty- 
five  AFIT  students.  These  subjects  can  be  assumed  to  be  a  representative  sample  of 
the  population  since  these  students  are  precisely  those  who  will  make  up  that  population 
in  future  years.  As  Air  Force  officers,  these  subjects  were  relatively  typical  of  the 
population  in  question,  possessing  similar  educational  and  professional  backgrounds. 

Variables 

The  variables  used  in  this  study  were  derived  from  the  author’s  study  of  the 
literature  surrounding  this  subject  area  and  the  subsequent  decision-making  model 
described  in  the  literature  review. 

Independent  Variable 

The  independent  variable  (Xi)  for  this  experiment  is  the  use  of  the  Expert  Choice 
computer-based  DSS.  The  subjects  were  introduced  to  the  capabilities  of  the  Expert 
Choice  software  and  provided  training  on  the  Analytical  Hierarchy  Process  on  which  the 
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DSS  is  based.  Each  subject  was  then  provided  Identical  computers  loaded  with  the 
Expert  Choice  software,  given  a  complex  decision  scenario,  and  tasked  with  selecting 
the  optimal  decision. 

Dependent  Variables 

The  following  dependant  variables  will  be  used  to  test  the  hypotheses: 

Time  to  Make  a  Decision.  The  actual  amount  of  time  required  to  reach  a 
decision.  This  was  measured  directly  in  this  experiment. 

Confidence  in  Decision.  A  measure  of  the  decision-maker’s  belief  that  the 
choice  selected  was  the  optimal  choice  available. 

Decision  Quality.  This  is  a  measurement  of  the  ability  to  select  the  “best” 
alternative  from  a  list  of  choices.  A  panel  of  experts  was  used  to  examine  the  decision 
scenario  and  select  the  “best”  alternative.  Since  the  decision  scenario  was  based  on  a 
source  selection  decision,  the  expert  panel  was  composed  of  individuals  experienced  in 
the  government  source  selection  decision  process.  The  subjects’  decisions  were 
compared  to  the  expert  panel’s  decision  and  if  they  were  the  same,  the  subject  made  a 
quality  decision. 

Number  of  Analyses  Considered.  This  is  a  measure  of  the  number  of  different 
ways  the  information  was  compared  prior  to  making  a  decision. 

Satisfaction  with  Decision  Process.  A  measure  of  the  decision-makers 
contentment  with  the  decision-making  process  used  to  make  the  decision. 
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Discussion  of  Hypotheses 


Chapter  11  discussed  the  decision-making  model  and  human  limitation  in  dealing 
with  information.  Due  to  these  limitations,  DSS  have  been  developed  to  aid  the 
decision-maker  in  reaching  as  rational  a  decision  as  possible.  The  goal  of  these  DSS 
systems  is  to  improve  the  decision  process.  In  this  experiment,  we  attempted  to 
determine  if  the  introduction  of  a  DSS  into  the  decision-making  process  would  impact 
the  dependent  variables. 

The  following  research  hypotheses  were  formulated: 

The  introduction  of  DSS  technoiogy  into  the  decision-making 
process  decreases  the  time  required  to  reach  a  decision. 

This  hypothesis  addresses  the  impact  of  using  DSS  technology  on  the  time 
required  for  the  decision-maker  to  reach  their  decision.  The  research  hypothesis  states 
that  using  DSS  will  not  affect  the  time.  It  is  important  to  determine  the  time  effects  since 
DSS  technology  could  be  disregarded  if  it  is  shown  to  delay  the  source  selection 
process. 

Hz  The  introduction  of  DSS  technoiogy  into  the  decision-making 
process  increases  the  confidence  with  the  decision. 

Confidence  addresses  the  belief,  after  reaching  a  dedsion,  that  the  decision¬ 
maker  made  the  right  decision.  The  confidence  that  the  decision-makers  feel  after 
reaching  a  decision  is  an  important  construct  because  the  decision-makers  must  be  able 
to  trust  their  DSS-aided  decisions. 
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Hj  The  introduction  of  DSS  technology  into  the  decision-making 
process  increases  decision  quality. 

This  hypothesis  addresses  the  impact  of  using  a  DSS  to  reach  a  decision  on  the 
decision’s  quality.  The  resultant  decision  is  the  most  important  part  of  the  decision 
process  and  the  effect  of  DSS  on  it  is  of  prime  importance. 

Previous  Results 

The  previous  study  gathered  data  using  a  questionnaire  with  a  five-point  Likert 
scale.  The  study  measured  the  level  of  agreement  or  disagreement  with  several 
decision  variables.  The  decision  variables  tested  included  decision  effectiveness, 
consistency,  speed,  difficulty,  understanding  and  confidence.  The  resultant  data  was 
condensed  into  ordinal  format  and  the  hypotheses  were  tested  using  Gamma  as  a 
correlation  coefficient.  Gamma  is  a  statistic  that  “compares  concordant  (P)  and 
discordant  (Q)  pairs  and  then  standardizes  the  outcome  by  maximizing  the  value  of  the 
denominator”  (Cooper  and  Emory,  1995:  506).  The  gamma  statistic  is  defined  as: 

y  =  (P-Q)/(P+Q) 

The  previous  study  reached  conclusions  using  statistical  tests  that  have  recently 
been  shown  to  be  less  than  optimal.  The  results  showed  that  DSS  usage  increased 
decision  time,  decreased  perceived  decision-making  difficulty,  increased  the 
understanding  of  the  decision-making  process,  and  increased  decision  confidence.  No 
significant  results  were  found  for  consistency  and  effectiveness  (Vickery,  1989:  53-55). 
This  study  seeks  to  gain  an  increased  insight  into  the  raw  data,  testing  the  hypotheses 
with  increased  statistical  tests.  The  goal  is  that  the  new  statistical  tests  will  allow  us  to 
more  accurately  test  the  hypotheses. 


42 


Hypothesis  Testing 

Now  that  Likert  scale  data  can  be  analyzed  as  interval  data,  the  means  by  which 

the  hypotheses  will  be  tested  will  change  greatly.  Each  hypothesis  will  be  tested  with 

the  analysis  of  variance  (ANOVA)  statistical  method  (Cooper  and  Emory,  1995: 457). 

This  method  tests  the  null  hypothesis  that  the  means  of  several  populations  are  equal. 

Cooper  and  Emory  describe  this  test  as  follows: 

(ANOVA)  uses  a  single-factor,  fixed-effects  model  to  compare  the 
effects  of  one  factor  on  a  continuous  dependent  variable.  To  use 
this  method  several  assumptions  must  be  met.  The  samples  must 
be  randomly  selected  from  normal  populations,  and  the  populations 
should  have  equal  variances.  ANOVA  is  reasonably  robust,  and 
minor  variations  from  normality  and  equal  variances  are  tolerable. 

(Cooperand  Emory,  1995: 457) 

The  test  statistic  for  the  ANOVA  compares  the  variances  and  is  called  the  F- 
ratio.  The  F-ratio  is  defined  as: 

F  =  Between-groups  variance  /  Within-groups  variance 

To  use  the  ANOVA  test,  several  assumptions  must  be  met.  The  “samples  must 
be  randomly  selected  from  normal  populations  and  the  populations  should  have  equal 
variances.  ANOVA  is  reasonably  robust,  and  minor  variations  from  normality  and  equal 
variances  are  tolerable”  (Cooper  and  Emory,  1995:  457). 

Interview 

In  order  to  gain  insight  into  the  perception  of  DSS  within  the  source  selection 
community,  inten/iews  were  conducted  with  several  individuals  closely  associated  with 
the  Air  Force’s  source  selection  process.  The  sample  of  source  selection  experts 
interviewed  for  this  study  was  a  convenience  sample.  The  subjects  were  presently 
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either  conducting  source  selections  or  involved  in  source  selection  planning/oversight  at 
Electronic  Systems  Center  (ESC),  Air  Force  Material  Command  (AFMC),  and 
Aeronautical  Systems  Center  (ASC).  The  interviews  were  conducted  over  the  telephone 
and  closely  followed  a  list  of  questions  centering  on  four  main  areas:  general  concerns 
with  the  source  selection  process,  experiences  with  DSS,  possibility  of  DSS 
implementation  in  the  source  selection  process,  and  perceptions  of  DSS  impact  on  the 
key  decisional  factors. 

The  telephone  interviewing  method  of  information  gathering  was  chosen  in  order 
to  contact  geographically  separated  individuals  and  acquire  the  “depth  and  detail  of 
information”  this  technique  affords  (Cooper  and  Emory,  1995:  271).  Cooper  and  Emory 
describe  three  main  conditions  that  must  be  met  to  have  a  successful  interview.  They 
are  “(1)  availability  of  the  needed  information  from  the  respondent,  (2)  an  understanding 
by  the  respondent  of  his  or  her  role,  and  (3)  adequate  motivation  by  the  respondent  to 
cooperate”  (Cooper  and  Emory,  1995:  271).  Of  these  conditions,  the  first  was  met  since 
the  questions  were  focused  on  attaining  an  understanding  of  the  respondents’  individual 
perceptions  and  these  perceptions  were  available  to  them  during  the  interview.  The 
second  condition  was  addressed  during  the  introduction  phase  of  the  interview.  In  this 
phase,  the  respondent  was  informed  of  the  two-phase  methodology  of  this  study  and 
where  their  inputs  would  be  used.  The  goal  of  this  phase  was  to  provide  the  respondent 
an  understanding  of  his  or  her  role  in  the  study.  The  third  condition  is  more  difficult  to 
address.  The  goal  of  the  introduction  phase  of  the  interview  was  to  detail  the  scope  of 
the  study  and  establish  a  friendly  relationship  with  the  respondent  in  the  hope  that  this 
would  motivate  the  cooperation  of  the  respondent. 
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Some  disadvantages  with  the  telephone  interviewing  method  include  a  limited 
interview  length  and  less  complete  responses.  The  limitation  on  interview  length  did  not 
apply  since  the  length  of  this  interview  (15-20  minutes)  fell  well  within  the  acceptable 
range  (Cooper  and  Emory,  1995:  281).  The  concern  with  less  complete  responses  was 
mitigated  by  the  interviewing  technique  used.  When  responses  were  incomplete,  the 
interviewer  used  probing  techniques  to  more  fully  and  relevantly  respond  to  the 
questions.  The  probing  techniques  used  included  expectant  pauses,  repeating  the 
question,  agreeing  with  the  respondent,  and  neutral  questions. 

To  begin  the  interview,  the  subjects  were  provided  a  brief  description  of  the 
research  effort.  Each  subject  was  given  a  description  of  a  computer-based  DSS  as 
described  in  the  literature  review  to  minimize  confusion  in  this  area.  The  positions  and 
office  symbols  of  each  interviewee  were  also  noted. 

Subjects  were  first  asked  to  explain  their  concerns  with  the  source  selection 
process  as  it  presently  exists  and  suggestions  as  to  how  they  would  improve  the 
deficiencies  they  defined.  These  questions  were  aimed  at  gaining  an  insight  into  the 
subjects’  general  perspectives  of  the  source  selection  process. 

Before  asking  specific  question  concerning  DSS,  each  individual  was  asked  to 
describe  any  personal  experience  with  a  DSS  in  the  source  selection  process.  This 
included  how  the  DSS  was  acquired,  the  impact  it  had  on  the  process,  and  other 
proposed  areas  for  implementation.  These  questions  address  the  present  level  of  DSS 
usage  within  the  source  selection  process. 

Each  subject  was  then  asked  whether  they  foresee  DSS  being  implemented  in 
the  source  selection  process.  They  were  also  questioned  as  to  where  in  the  process  the 
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DSS  should  be  implemented  with  a  focus  on  whether  they  personally  are  open  to  the 
possibility  of  using  a  DSS  within  their  job.  The  emphasis  of  this  section  was  to  gauge 
their  perception  of  future  DSS  implementation. 

Finally,  each  subject  was  asked  to  predict  the  impact  of  DSS  usage  on  the 
decisional  factors  of  decision  time,  alternatives  considered,  confidence,  satisfaction,  and 
quality.  The  subjects  were  asked  to  accomplish  this  for  both  a  DSS  used  by  the  SSA 
and  a  DSS  used  in  lower  process  levels.  The  choices  for  each  decision  factor  were 
limited  to  increase,  decrease,  or  remain  the  same.  This  quantified  their  perceptions  of 
how  DSS  usage  would  affect  the  source  selection  process  at  both  levels.  The  interview 
questionnaire  used  in  this  study  is  included  in  the  appendix. 

Proposition  Evaluation 

The  data  gained  through  interviews  will  address  the  propositions.  These 
propositions  will  not  be  tested  with  statistical  tests  but  will  be  addressed  with  a 
“preponderance  of  evidence”  test.  The  following  propositions  will  be  addressed  using 
the  interview  data: 

Pi  The  introduction  of  DSS  technology  into  the  decision-making 
process  increases  the  number  of  ways  the  information  was 
analyzed. 

The  literature  has  shown  that  the  number  of  decision  analyses  conducted  is  positively 
correlated  to  the  quality  of  the  ultimate  decision.  This  proposition  compares  the  number 
of  decision  analyses  considered  in  the  decision-making  process  with  and  without  DSS 
use. 
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Pj  The  introduction  of  DSS  technoiogy  into  the  decision-making 
process  increases  satisfaction  with  the  decision  process. 

This  proposition  addresses  the  satisfaction  the  decision-makers  feel  with  the  decision 
process  and  whether  the  satisfaction  will  differ  between  individuals  using  the  DSS  and 
those  not  using  the  DSS. 

P3  The  perception  of  DSS  technology  within  the  source  seiection 
community  has  improved  within  the  last  five  years. 

Computer-based  DSS  technologies  have  met  with  resistance  in  the  past.  With  the 
computer’s  acceptance  as  an  integral  part  of  the  work  environment,  an  improved 
perception  of  DSS  technology  was  proposed.  An  improving  perception  of  the 
technology  within  the  source  selection  community  is  important  if  the  technology  is  to  be 
implemented. 

P4  The  source  selection  community  foresees  DSS  technoiogy  being 
implemented  within  the  decision  process. 

It  is  important  to  determine  whether  the  source  selection  community  believes  this 
technology  has  a  future  in  the  process.  This  proposition  addresses  whether  individuals 
involved  in  the  process  are  of  the  opinion  that  this  technology  will  be  a  part  of  the 
process  in  the  future. 

Conclusion 

The  two-fold  methodology  described  in  this  section  details  how  the  hypotheses 
and  propositions  will  be  addressed.  The  experimental  data  gained  from  the  previous 
study  will  be  used  to  test  hypotheses  one  through  three.  This  will  be  accomplished 
using  the  ANOVA  statistical  test.  The  interview  data,  gathered  from  individuals 
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presently  working  within  the  source  selection  process,  will  be  used  to  address 
propositions  one  through  four.  The  goal  of  using  the  combination  of  experimental  and 
interview  data  in  this  study  is  to  address  the  hypotheses  and  propositions  and  gain 
insight  into  the  perceptions  of  DSS  within  the  source  selection  community.  The 
following  chart  details  how  the  hypotheses  will  be  tested; 


Table  2.  Hypothesis/Proposition  Test  Methods 


Hypotheses  and  Propositions 

Test  Method 

H1 .  DSS  usage  decreases  the  time  required  to  make  a 
decision 

Empirical  data  from 
experiment  (ANOVA) 

H2.  DSS  use  increases  the  confidence  in  the  decision 

Empirical  data  from 
experiment  (ANOVA) 

H3.  DSS  use  increases  decision  quality 

Empirical  data  from 
experiment  (ANOVA) 

P1 .  DSS  use  increases  the  number  of  ways  the 
information  was  analyzed  prior  to  reaching  a  decision 

Interview  data 

P2.  DSS  use  increases  the  satisfaction  with  the  decision 
process 

interview  data 

P3.  The  perception  of  DSS  technology  within  the  source 
selection  community  has  improved  within  the  last  five 
years 

interview  data 

P4.  The  source  selection  community  foresees  DSS 
technology  being  implemented  within  the  decision 
process 

Interview  data 

The  analysis  of  these  hypotheses  and  propositions  will  provide  important 
information  concerning  the  viability  of  implementing  DSS  technology  within  the 
contracting  source  selection  process.  The  three  hypotheses  address  the  impact  of  DSS 
usage  on  key  decisional  factors  using  empirical  data  taken  from  a  previous  study. 
Propositions  one  and  two  address  the  perceived  impact  of  DSS  implementation  on  other 
decisional  factors.  Finally,  propositions  three  and  four  address  the  perception  and 
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future  of  DSS  technology  within  the  source  selection  community.  The  next  chapter  will 
describe  the  results  of  addressing  these  hypotheses  and  propositions  and  describe  the 
other  information  gathered  from  the  interview  process. 
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IV.  Analysis 


Introduction 

This  chapter  will  focus  on  the  testing  of  the  hypotheses  and  propositions 
described  in  this  study  and  the  reporting  of  free-form  data  gathered  from  the  inten/iew 
process.  The  Hypotheses  section  will  address  the  information  gathering  and  the  results 
of  testing  the  three  hypotheses  and  four  propositions  described  in  the  Methodology 
section.  The  Free-Form  Data  section  will  discuss  other  information  gained  from  the 
interview  process. 

Findings 

This  study  utilizes  a  two-part  methodology  to  address  the  hypotheses  and 
propositions.  The  first  three  hypotheses  address  time,  confidence,  and  decision  quality 
and  will  be  analyzed  using  empirical  data  from  a  previous  study.  Propositions  one 
through  four  will  be  addressed  using  interview  data  gathered  for  this  study. 

Time  to  Make  a  Decision 

Hypothesis  1 .  The  introduction  of  DSS  technology  into  the  decision-making 
process  decreases  the  time  required  to  reach  a  decision. 

By  measuring  the  number  of  minutes  each  decision-maker  required  to  reach 
their  decision,  an  objective  measure  of  time  was  attained.  Figure  4  depicts  the  results  of 
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the  control  and  experimental  groups  with  respect  to  time.  The  figure  shows  that  the 
control  group  seemed  to  require  less  time  to  make  a  decision  than  the  experimental 
group. 

The  ANOVA  statistical  test  was  used  to  test  hypothesis  number  1.  This 
hypothesis  addresses  whether  individuals  using  DSS  technology  will  require  less  time  to 
make  a  decision  when  compare  to  decision-makers  not  using  DSS  technology. 


Time  in  Minutes 
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Figure  4.  Distribution  of  Time  in  Minutes 

This  research  hypothesis  is  not  supported.  The  statistical  analysis  detailed 
below  shows  that  the  average  time  for  the  control  group  was  108.96  minutes  while  the 
average  time  for  the  experimental  group  was  140.00  minutes.  The  ANOVA  test  was 
used  to  determine  if  this  difference  is  statistically  significant.  When  these  two  groups 
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are  tested  according  to  time,  the  results  clearly  show  that  the  means  are  not  equal  (F  > 
Fcrit)  and  that  the  experimental  (DSS)  group  required  more  time  than  the  control  group 
to  reach  a  decision. 


Tables.  ANOVA  Statistics  for  Time 
ANOVA:  Single  Factor  SPEED 


SUMMARY 


Groups 

Count  Sum 

Average 

Variance 

Control 

26  2833 

108.96154  483.47846 

Experimental 

19  2660 

140 

1185.8889 

ANOVA 

Source  of 
Variation 

SS 

df 

MS 

F  P-value 

Fcrit 

Between 

Groups 

10575.838 

1 

10575.838 

13.602177  0.0006301 

4.0670471 

Within 

Groups 

33432.962 

43 

777.51073 

Total 

44008.8 

44 

This  research  hypothesis  is  not  supported  since  the  information  above  shows 
that  DSS  usage  actually  increased  the  time  required  to  reach  a  decision. 


Confidence  in  Decision 

Hypothesis  2.  The  introduction  of  DSS  technology  into  the  decision-making 
process  increases  the  confidence  with  the  decision. 

Confidence  was  measured  using  a  five-point  Likert  scale  questionnaire.  The 
statement  “I  am  confident  that  I  made  the  right  decision”  was  used  to  address  this 
dependent  variable.  Neutral  responses  were  discarded.  Strongly  agree  and  mildly 
agree  responses  were  combined  into  a  single  agree  category.  The  same  was  done  for 
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the  disagree  responses.  As  seen  in  Figure  5,  there  is  a  slight  difference  between  the 
two  groups  which  indicates  an  increase  in  confidence  with  the  experimental  group. 


Figure  5.  Distribution  of  Confidence  Level 

Hypothesis  two  was  also  analyzed  using  the  ANOVA  statistical  test.  This 
hypothesis  addresses  the  confidence  the  decision-makers  had  that  they  made  the  right 
decision  and  whether  this  differed  between  the  control  and  experimental  groups. 

This  research  hypothesis  Is  not  supported.  The  average  value  for  the  control 
and  experimental  groups  were  2.27  and  2.16  respectively.  These  numbers  correspond 
to  the  five-point  Likert  scale  with  lower  numbers  relating  to  a  stronger  feeling  of 
confidence.  When  these  means  are  tested  with  the  ANOVA  test  statistic,  no  statistically 
significant  difference  between  the  two  groups  can  be  found  (F<Fcrit). 
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Table  4.  ANOVA  Statistics  for  Confidence 


Anova:  Single  Factor 

Confidence 

SUMMARY 

Groups 

Count 

Sum 

Average 

Variance 

confidence 

19 

22 

1.157895 

0.140351 

confidence 

17 

1.176471 

0.154412 

ANOVA 


Source  of  Variation 

SS 

df 

MS  F  P-vaiue  F  crit 

Between  Groups 

0.003096 

1 

0.003096  0.021066  0.885457  4.130015 

Within  Groups 

4.996904 

34 

0.146968 

Total 

5 

35 

Decision  Quality 

Hypothesis  3.  The  introduction  of  DSS  technology  into  the  decision-making 
process  increases  decision  quality. 

Decision  quality  was  measured  objectively  by  comparing  the  decision  reached 
with  the  correct  answer.  Figure  6  shows  the  strong  similarity  in  the  decisions  reached 
by  the  control  and  experimental  groups.  The  “correct”  answer  for  this  decision 
scenario,  as  determined  by  a  panel  of  experts,  was  the  selection  of  Magnetic  Tech. 

Hypothesis  three  was  also  tested  using  the  ANOVA  statistical  test.  This 
hypothesis  addresses  the  difference  between  the  control  and  experimental  groups’ 
ability  to  select  the  “best”  alternative. 
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Contractor  Choice 


i 


in  Control 
i  B  Experimental ! 


Figure  6.  Distribution  of  Decision  Choice 


This  research  hypothesis  is  not  supported.  The  means  of  the  control  and 
experimental  groups  are  nearly  identical  and  the  ANOVA  test  fails  to  show  a  significant 
difference  between  the  two  values  (F<Fcrit). 


Table  5.  ANOVA  Statistics  for  Decision  Quality 


Anova;  Single  Factor  Quality 

SUMMARY 

Groups 

Count 

Sum  Average  Variance 

Control 

Experimental 

26 

19 

34  1.3076923  0.6215385 
24  1.2631579  0.4269006 

ANOVA 

Source  of  Variation 

SS 

df  MS  F  P-value  F  crit 

Between  Groups 

0.0217724 

1  0.0217724  0.0403146  0.8418135  4.0670471 

Within  Groups 

23.222672 

43  0.5400621 

Total 

23.244444 

44 
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Decision  Analyses  Considered 


Proposition  1.  This  proposition  addresses  a  perception  of  DSS  within  the  source 
selection  community.  Research  seems  to  suggest  that  the  more  ways  the  information 
pertaining  to  a  decision  is  analyzed,  the  better  the  resultant  decision  will  be  (Cats-Baril, 
1997:  368).  Because  of  this  relationship,  this  was  included  as  a  proposition  in  this 
study.  Lacking  empirical  data  in  this  area,  the  interview  process  was  used  to  gather  the 
data  required.  Each  interviewee  was  provided  a  description  of  the  capabilities  and 
limitations  of  DSS  and  asked  whether  they  believed  that  the  use  of  such  technology 
would  have  an  impact  on  the  “number  of  ways  the  information  is  analyzed  prior  to 
reaching  a  decision.”  The  respondents  were  allowed  only  to  respond  with  increase, 
decrease,  or  no  change. 

Pi  The  introduction  of  DSS  technology  into  the  decision-making  process  will 
increase  the  number  of  ways  the  information  was  analyzed 

This  proposition  is  not  supported  by  interview  results.  63%  of  the  interviewees 
believed  that  the  introduction  of  DSS  technology  into  the  source  selection  process  would 
have  no  impact  on  the  number  of  ways  the  information  would  be  analyzed  prior  to 
reaching  a  decision.  The  results  are  depicted  in  Figure  7. 
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impact  of  DSS  on  the  Number  of 
Ways  Information  is  Analyzed  Prior 
to  Reaching  a  Decision 
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Figure  7.  Distribution  of  Change  in  the  Number  of  Ways  Information  is  Analyzed  with 

DSS  Use 


Satisfaction  with  Decision 

Proposition  2.  This  proposition  addresses  the  satisfaction  the  decision-makers 
feel  with  the  decision  process  and  whether  this  will  change  with  the  introduction  of  DSS 
technology.  This  proposition  was  also  addressed  using  interview  data  gathered  from 
experts  within  the  source  selection  process.  They  were  asked  whether  they  believed 
the  satisfaction  with  the  decision  process  would  increase,  decrease,  or  remain  the  same 
with  DSS  use. 

P2  The  introduction  of  DSS  technology  into  the  decision-making  process  will 
increase  the  satisfaction  with  the  decision  process. 
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This  proposition  is  not  supported  by  the  interview  responses.  The  interview 
responses  were  evenly  distributed  between  “no  effect”  and  “increase.”  The  interview 
results  are  inconclusive  and  thus  we  cannot  support  the  proposition.  The  response 
distribution  is  displayed  in  Figure  8. 


Impact  of  DSS  on  the  Satisfaction  with 
the  Decision  Process 
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Figure  8.  Distribution  of  Decision  Satisfaction 

Perception  of  DSS  Technology 

Proposition  3.  Computer-based  DSS  technologies  have,  in  the  past,  met  with 
resistance  by  certain  employees.  Now  that  computer-based  technology  has  become 
commonplace  in  the  work  environment  within  the  last  five  years,  an  improved  perception 
of  DSS  technology  within  the  source  selection  community  was  proposed.  Interview  data 
was  used  to  address  this  proposition.  Individuals  working  within  the  source  selection 
process  were  asked  whether  they  believed  the  “perception  of  DSS  technology  within  the 
source  selection  community  has  improved  within  the  last  five  years.” 
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Pj  The  perception  of  DSS  technology  within  the  source  selection  community 
has  improved  within  the  last  five  years. 

The  interview  data  supports  this  proposition.  75%  of  the  interviewees  stated  that 
the  perceptions  of  DSS  within  the  source  selection  community  have  improved  within  the 
last  five  years  while  the  other  25%  stated  that  there  was  no  change.  The  results  are 
depicted  in  Figure  9. 


The  Change  in  Perception  of  DSS  Within 
the  Last  Five  Years 


Improved  No  Change  Worsened 

Interview  Responses 


Figure  9.  Distribution  of  Change  in  DSS  Perception 


DSS  Implementation 

Proposition  4.  Eventual  introduction  of  DSS  technology  within  the  source 
selection  process  is  dependent  largely  upon  the  very  individuals  interviewed  for  this 
study,  the  professionals  working  in  the  planning  and  actual  decision-making  process.  It 
was  proposed  that  the  interviewees  would  foresee  DSS  technology  being  implemented 
in  the  source  selection  process. 
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P4  The  source  selection  community  foresees  DSS  technology  being 
implemented  within  the  decision  process. 

The  interview  data  supports  this  research  proposition.  The  interviewees  almost 
unanimously  believed  that  DSS  technology  would  be  implemented  within  the  source 
selection  process.  Though  many  of  the  respondents  mentioned  different  areas  for  DSS 
implementation  and  different  benefits  of  such  implementation,  the  great  majority 
foresees  DSS  technology  being  used  in  the  decision  process. 

The  interview  data  supports  the  proposition  that  DSS  technology  will  be 
implemented  in  the  process.  88%  of  the  interviewees  stated  that  they  foresee  DSS 
technology  being  implemented  in  the  source  selection  process.  The  distribution  of 
interview  responses  is  depicted  in  Figure  10. 


Do  You  Foresee  DSS  Implementation  into 
the  Source  Selection  Process? 


Interview  Responses 


Figure  10.  Distribution  of  Foreseen  DSS  Implementation 
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Free-Form  Data 


This  section  will  address  other  questions  asked  in  the  interviews  of  individuals  involved 
in  the  source  selection  process. 

1 .  What  are  your  present  concerns  with  the  source  selection  process? 

The  most  common  responses  to  this  question  had  to  do  with  the  complexity  of 
the  process,  the  time  it  takes,  and  the  formation  of  teams.  The  complexity  of  the 
process  prevented  the  customer/buyer  from  understanding  what  was  involved  and  had  a 
large  impact  on  the  time  required  to  complete  the  process.  As  service  providers,  the 
interviewers  felt  that  their  process  could  be  simplified  and  therefore  sped  up.  That  is 
what  the  customers  are  looking  for,  a  quality  product  delivered  quickly.  The  personnel 
issue  was  also  mentioned  by  about  75%  of  the  interviewees.  They  emphasized  the 
importance  of  quality  people  being  involved.  They  stated  their  frustration  with  the  large 
team  sizes  and  high  turnover  rates.  The  interviewees  believed  that  the  combination  of 
these  effects  led  to  teams  being  composed  of  individuals  who  are  untrained  and 
unfamiliar  with  the  process,  further  delaying  the  process. 

2.  What  can  be  done  to  improve  the  source  selection  process? 

As  with  the  question  above,  the  responses  concentrated  on  simplifying  the 
process  and  improving  the  training.  About  40%  of  the  interviewees  stated  that  the 
number  of  criteria  examined  for  each  source  selection  could  be  reduced  in  order  to 
focus  attention  on  those  few  “discriminators.”  Training  continued  to  be  a  area  of 
concern  for  the  interviewees.  A  pervasive  concern  mentioned  by  the  interviewees  was 
that  teams  were  being  compiled  of  untrained  members  and  this  was  reducing  their 
effectiveness  and  efficiency.  They  believed  a  training  program  that  is  more  flexible  and 
implemented  with  more  pre-planning  would  benefit  the  process. 
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3.  Are  you  presently  using  a  DSS  within  the  source  selection  process? 

No  interviewee  was  presently  using  a  formal  DSS  specifically  designed  for  the 
source  selection  process.  They  were  all  using  basic  computer  software  (Microsoft 
Excel)  in  the  process  to  capture  some  of  the  benefits  of  DSS  technology  to  include  the 
compiling  and  displaying  of  large  amounts  of  information.  Several  interviewees 
mentioned  that  they  had  received  training  on  a  commercial-off-the-shelf  Group  DSS 
system  that  was  planned  for  implementation  in  the  near  future.  This  system  was 
procured  to  improve  the  consensus  building  between  group  members  and  aid  in  the 
documentation  of  the  process. 

4.  What  do  foresee  as  the  impact  of  using  DSS  technology  within  the  source  selection 
process? 

The  answers  ranged  greatly  to  this  question.  Some  respondents  felt  that  the 
DSS  would  create  an  over  reliance  on  “hard”  data  and  reduce  the  discussion  between 
decision-makers.  Others  felt  the  capabilities  of  DSS  technology  would  improve  the 
process  by  allowing  the  decision-maker  to  access  great  amounts  of  lower-level,  detailed 
information  when  needed.  Several  interviewees  also  mentioned  that  a  Group  DSS 
would  facilitate  consensus  building  within  groups  involved  in  the  process. 

5.  Where  in  the  process  do  you  foresee  DSS  technology  being  used? 

Over  80%  of  the  interviewees  believed  DSS  technology  would  best  be  applied  at 
the  lower  levels  within  the  source  selection  process.  They  believed  that  though  the 
model  building,  information  retrieval,  and  display  capabilities  could  benefit  the  SSA,  the 
real  benefits  would  be  seen  in  its  implementation  within  the  technical  and  contractual 
teams.  GDSS  technology  was  often  mentioned  as  a  beneficial  DSS  technology  for 
building  consensus  within  the  separate  technical  expertise  areas. 
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6.  What  do  you  foresee  being  lost  with  DSS  implementation  within  the  source  selection 
process? 

About  half  the  interviewees  believed  that  nothing  would  be  lost  if  DSS  technology 
was  implemented  in  the  process.  The  other  half  mentioned  concerns  of  lost  subjectivity. 
There  seemed  to  be  a  fear  that  the  DSS  would  have  a  “magic  formula”  that  would 
provide  the  right  answer  to  the  decision-maker.  The  fear  was  that  subjective  areas 
would  be  ignored  with  the  emphasis  on  numerical  data  and  that  the  decision-maker 
would  lose  his/her  power. 

Conclusion 

Of  the  seven  research  hypotheses  and  propositions  addressed  in  this  study,  the 
data  supported  only  two.  Propositions  three  and  four  were  supported  by  the  data. 

Table  6  summarizes  the  results  of  the  analysis  in  this  study. 
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Table  6.  Results  of  Hypothesis/Proposition  Testing 


Hypotheses/Propositions 

Test  Method 

Results 

HI.  DSS  usage  decreases  the  time 
required  to  make  a  decision 

Empirical  data  from 
experiment  (ANOVA) 

Not  Supported 

H2.  DSS  use  increases  the  confidence 
in  the  decision 

Empirical  data  from 
experiment  (ANOVA) 

Not  Supported 

H3.  DSS  use  increases  decision 
quality 

Empirical  data  from 
experiment  (ANOVA) 

Not  Supported 

PI .  DSS  use  increases  the  number  of 
ways  the  information  was  analyzed 
prior  to  reaching  a  decision 

Interview  data 

Not  Supported 

P2.  DSS  use  increases  the  satisfaction 
with  the  decision  process 

Interview  data 

Not  Supported 

P3.  The  perception  of  DSS  technology 
within  the  source  selection  community 
has  improved  within  the  last  five  years 

Interview  data 

Supported 

P4.  The  source  selection  community 
foresees  DSS  technology  being 
implemented  within  the  decision 
process 

Interview  data 

Supported 

Though  several  research  hypotheses/propositions  were  not  proven,  two 
important  propositions  were  supported.  The  interviewees  believed  that  the  perception  of 
DSS  technology  has  improved  within  the  last  five  years.  They  also  stated  their  belief 
that  DSS  technology  will  be  implemented  within  the  source  selection  process.  The  free¬ 
form  data  gained  from  the  interview  process  also  provided  many  interesting  points 
concerning  the  use  of  DSS  in  the  source  selection  process.  These  will  be  discussed 
further  in  the  following  chapter. 
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V.  Conclusions  and  Recommendations 


Introduction 

In  the  conclusion,  a  brief  summary  of  the  results  found  in  chapter  IV  will  be 
presented  with  an  added  emphasis  on  the  importance  of  these  findings.  From  the 
information  discussed  in  the  Conclusions  section,  suggestions  will  be  made  in  the 
Recommendations  section  for  further  research  in  this  subject  area. 

Results 

The  two-part  methodology  employed  in  this  study  provided  information  ranging 
from  empirical  data  (experimental)  to  personal  conversations  with  individuals  involved  in 
the  source  selection  process  (interviews).  The  results  of  the  analysis  of  the  empirical 
data  were  disappointing  to  the  author.  The  inability  to  show  any  positive  relationship 
between  DSS  usage  and  the  constructs  of  time,  confidence  and  decision  quality 
concurred  with  the  previous  study  (Vickery,  1989)  but  differed  from  the  consensus  of 
literature  discussed  in  Chapter  ll.  The  statistical  tests  employed  in  this  study  did  not 
support  the  research  hypotheses  dealing  with  time,  confidence,  and  decision  quality. 

The  difficulty  in  showing  the  hypothesized  relationships  could  be  caused  by  the 
experiment  itself.  Though  time  to  make  a  decision  actually  increased  with  DSS  use,  this 
could  be  explained  by  the  learning  curve.  The  subjects  used  in  this  study  were 
inexperienced  with  the  DSS  software  and  the  added  time  they  required  to  reach  a 
decision  could  be  due  their  inexperience  using  the  software  and  not  due  to  the  DSS 
itself.  The  measurement  of  decision  time  after  the  subjects  had  experience  with  the 
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DSS  and  were  comfortable  with  it  could  lead  to  an  improved  isolation  of  the  effect  of 
DSS  use  on  decision  time  by  effectively  removing  the  learning  curve  effect.  The 
literature  demonstrated  this  proposed  “learning  curve”  effect  with  respect  to  DSS. 
Studies  have  shown  differences  between  DSS  users  and  non-DSS  users  disappear 
after  several  trials  (Sharda,  1988;  153),  (MacKay,  1992:  660-663). 

The  research  hypothesis  addressing  levels  of  confidence  was  also  not 
supported.  The  levels  of  confidence  for  both  the  experimental  and  control  groups  were 
nearly  identical.  Though  the  experimental  group,  using  the  DSS,  did  show  higher  levels 
of  confidence,  the  difference  was  not  shown  to  be  statistically  significant.  This 
corresponds  to  the  slight  positive  effect  evidenced  in  the  literature  (Cats-Baril  and 
Huber,  1987:  370),  (Sharda,  1988:154). 

The  research  hypothesis  with  respect  to  decision  quality  was  also  not  supported. 
One  possible  explanation  for  these  results  could  be  due  to  the  relative  simplicity  of  the 
decision  scenario.  Over  80%  of  both  the  control  and  experimental  decision-makers 
selected  the  optimal  decision.  The  introduction  of  a  more  difficult  decision  scenario 
could  spread  the  data  and  perhaps  show  some  difference  between  the  two  groups.  The 
decision  scenario,  though  modeled  after  a  source  selection  decision  scenario,  lacked 
the  complexity  that  is  involved  in  actual  source  selection  decisions. 

The  interview  data  provided  unique  insight  into  the  perceptions  of  individuals 
involved  in  the  source  selection  community.  Though  the  interviewees  were  far  from  a 
representative  sample  of  everyone  involved  in  the  source  selection  arena,  they  were 
individuals  involved  in  the  planning  and  oversight  of  the  source  selection  process  at  the 
major  command  and  system  center  level.  The  perceptions  of  these  individuals  with 
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respect  to  DSS  will  greatly  influence  Its  possibility  for  implementation  in  the  process. 

The  interview  data  addressed  the  four  propositions  and  also  provided  free-form  data 
that  proved  significant. 

Two  research  propositions  were  supported  by  the  Interview  data.  The  perception 
of  DSS  technology  within  the  source  selection  community  was  shown  to  have  improved 
in  the  last  five  years.  The  interviewees  repeatedly  stated  that  the  increased  reliance  on 
the  desktop  computer  within  the  office  environment  has  removed  many,  but  not  all, 
reservations  and  made  the  introduction  of  DSS  technology  more  “acceptable.”  One 
interviewee  stated  that  “DSS  is  no  longer  a  foreign  concept  -  there  is  less  of  a  belief  that 
the  computer  is  intruding  on  the  work  environment.”  Another  interviewee  stated  that 
“people  are  more  used  to  using  computers  to  assist  them  in  making  decisions”  and  that 
this  has  “led  to  DSS  implementation.”  This  seems  to  suggest  that  the  source  selection 
community  is  becoming  more  open  to  DSS  usage  within  the  decision  process. 

The  other  research  proposition  supported  by  the  interview  data  dealt  with  the 
whether  these  source  selection  experts  foresaw  DSS  technology  being  implemented 
within  the  decision  process.  The  interviewees  supported  the  proposition  that  DSS 
technology  will  be  implemented  within  the  source  selection  process.  Though  the  great 
majority  of  interviewees  agreed  that  DSS  would  be  implemented,  they  contradicted  the 
author’s  belief  as  to  where  in  the  source  selection  process  the  DSS  would  provide  the 
best  utility.  While  the  author  believed  that  DSS  would  provide  the  greatest  benefit  to  the 
SSA  in  the  final  decision  between  contractor’s  proposals,  the  intenriewees  almost 
unanimously  believed  that  DSS  technology  would  provide  more  utility  to  the  lower  levels 
of  the  decision  process.  They  felt  instead  of  a  DSS,  that  a  group  decision  support 
system  (GDSS)  should  be  used  by  the  separate  technical,  contracting  and  managerial 
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teams  (SSET)  to  facilitate  their  assessment  of  the  proposals  prior  to  forwarding  their 
evaluations  to  the  SSA.  This  contradicted  a  key  assumption  of  the  author,  namely  that 
the  SSA  would  use  the  DSS  in  the  final  decision.  The  interviewees  repeatedly 
mentioned  that  a  GDSS  would  assist  in  the  “reaching  of  consensus,”  the 
communication,  and  the  documentation  within  these  specialized  teams. 

Summary 

“Making  adequate  decisions  over  long  time  periods  in  a  changing  environment 
and  subject  to  incomplete  information,  misinformation,  uncertainty,  and  changing 
preferences  is  one  of  the  central  and  most  sophisticated  human  cognitive  abilities” 
(Radermacher,  1994:  258).  How  we  facilitate  this  decision-making  process  within  the 
source  selection  arena  is  the  focus  of  this  study.  Though  the  results  failed  to  show  the 
effects  of  DSS  use  on  key  decisional  factors,  valid  conclusions  can  still  be  drawn  from 
this  study.  Individuals  involved  in  the  source  selection  process  have  an  improving 
perception  of  DSS  and  foresee  its  implementation  within  the  decision-making  process. 
They  believe  that  GDSS  technology  should  be  implemented  within  the  lower  levels  of 
the  source  selection  organization  (SSET). 

The  workplace  is  becoming  increasingly  interconnected  via  information 
technology  such  as  the  internet,  LAN,  intranet,  etc.  This  technology  of  connecting 
computers  together  has  shown  to  be  advantageous  to  group  decision  making  when 
used  in  conjunction  with  GDSS  technology  (McLeod  1997:  714).  A  study  conducted  by 
Gallupe  et  al.  found  that  decision  quality  for  groups  using  the  GDSS  was  significantly 
greater  than  groups  not  using  the  system(Gallupe  et  al.,  1988:  293).  Gallupe  et  al.  also 
noted  that  the  difference  in  decision  quality  increased  as  task  difficulty  increased. 
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providing  evidence  that  GDSS’s  may  provide  the  greatest  value-added  for  difficult  tasks 
(Gallupe  et  al.  1988:  277).  Another  study  by  Zigurs  et  al.  also  found  that  “the  procedural 
efforts  of  CS  [GDSS]  groups  were  more  effective  than  those  of  manual  groups”  and  that 
the  GDSS  groups  outperformed  the  manual  groups  (Zigurs  et  al.  1988:  641). 

McLeod  conducted  another  study  of  the  benefits  of  GDSS  use.  This  study 
stated  that  GDSS  groups  outperform  face-to-face  groups  by  reducing  barriers  to 
participation  (McLeod  et  al.,  1997:  706-707).  Reasons  for  this  improved  performance 
are  that  GDSS  technology  eliminates  the  need  for  turn-taking,  reduces  apprehension  to 
participation,  and  reliance  on  anonymity  (McLeod  et  al.,  1997:  707).  The  McLeod  study 
found  a  relationship  between  the  reduction  in  barriers  to  participation  found  in  GDSS  to 
overall  group  decision  quality  (McLeod  1997:  714).  This  study  bases  the  improved 
group  performance  with  GDSS  technology  on  the  improved  participation  of  all  group 
members  in  the  GDSS  configuration.  Some  group  members  who  othenvise  contribute 
little  to  group  consensus  building  are  more  eager  to  contribute  in  the  GDSS  mode.  The 
overall  consensus  within  the  literature  is  that  GDSS  technology  improves  decision 
quality,  participation,  and  satisfaction  (Eierman  et  al.,  1995:  8-9). 

The  consensus  of  the  interviewees  was  that  GDSS  technology  would  provide  the 
greatest  benefit  to  the  source  selection  process  and  that  the  lower  levels,  namely  the 
technical  and  contractual  teams  assessing  the  proposals  prior  to  SSA,  should  utilize  this 
technology.  The  perceived  usefulness  of  GDSS  technology  by  the  interviewees  is 
largely  supported  in  the  literature.  The  documented  benefits  of  GDSS,  along  with  the 
improved  perception  of  DSS  technology,  point  to  a  merging  of  technology  and  need 
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within  the  source  selection  community.  The  utilization  of  GDSS  technology  within  the 
source  selection  process  has  serious  consequences  to  future  source  selections  that 
must  be  examined  further. 

Barriers  to  Technology 

Though  source  selection  community  perceptions  of  DSS  technology  have 
improved  and  they  foresee  its  ultimate  implementation,  little  has  been  done  to  actually 
bring  this  technology  into  this  process.  There  must  be  some  explanation  for  this.  Since 
the  source  selection  community,  or  at  least  those  interviewed  for  this  study,  see  the 
benefits  of  this  technology  but  yet  little  has  been  done,  there  seems  to  be  some 
organizational  resistance.  The  theory  of  innovation  diffusion  attempts  to  explain  this 
resistance  to  acceptance  of  information  technology. 

Innovation  diffusion  theory  attempts  to  describe  how  any  technological 
innovation  moves  from  the  stage  of  invention  to  widespread  use.  This  theory  posits  five 
characteristics  beneficial  to  the  eventual  widespread  use;  relative  advantage, 
compatibility,  complexity,  trialability,  and  observability  of  outputs  (Dillon  and  Morris, 

1 996:  6).  Of  these  five  characteristics,  compatibility,  complexity,  and  observability  could 
be  precluding  the  adoption  of  DSS.  DSS  seems  compatible  with  the  social  practices 
and  norms  of  those  interviewed  for  this  study  but  the  power  base  in  the  source  selection 
community  is  with  the  SSA’s  themselves  and  this  technology  may  threaten  their  position 
and  power.  The  complexity  of  DSS  technology  is  another  barrier  for  innovation 
diffusion.  Though  the  technology  was  perceived  favorably  by  most  source  selection 
individuals,  there  was  some  resistance  expressed  in  the  interviews  to  the  increased 
complexity  of  the  process  and  the  DSS  technology  itself.  Observability  describes  “the 
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extent  to  which  a  technology’s  output  and  its  gains  are  clear”  (Dillon  and  Morris,  1996: 

6).  This  presents  another  problem  since  with  source  selection  decisions  it  is  extremely 
difficult  to  quantify  the  improvement  in  selecting  one  proposal  over  another. 

Attewell’s  view  of  innovation  diffusion  is  that  the  burden  of  developing  a  technical 
know-how  becomes  the  most  significant  hurdle  to  adoption  (Attewell,  1992:  7). 

According  to  his  view,  the  delay  we  are  seeing  in  DSS  implementation  could  simply  be 
the  result  of  the  time  needed  to  develop  the  technical  “organizational  learning  required 
to  implement  and  operate  it  successfully”  (Attewell,  1992: 7). 

The  explanation  for  why  this  available  and  seemingly  productive  technology  has 
yet  to  be  introduced  into  the  source  selection  community  is  multi-faceted.  The  delay  can 
be  explained  by  a  combination  of  lacking  organizational  know-how  and  threats  to  power 
and  position.  Simply  put,  there  is  little  corporate  experience  with  developing, 
implementing,  and  operating  decision  support  system  technology  within  the  Air  Force. 
The  development  of  an  organizational  knowledge  base  to  inform  the  source  selection 
community  about  the  capabilities,  demonstrate,  and  build  support  for  eventual 
implementation  is  crucial.  This  group  could  ensure  that  the  five  characteristics  of 
innovation  diffusion  theory  are  applied. 

The  final  and  biggest  hurdle  to  DSS  implementation  is  that  it  is  perceived  as  a 
threat  to  individual  power  and  position.  The  information  gathered  from  the  interview 
process  seems  to  indicate  a  strong  concern  for  how  this  technology  will  impact  SSAs. 
They  seem  to  protest  the  introduction  of  DSS  technology  not  because  of  its  capabilities 
or  limitations  but  because  of  a  perceived  reduction  in  the  subjectivity  of  their  decision¬ 
making.  DSS  technology  provides  organized,  clear  information  along  with  the  ability  to 
manipulate,  compare,  and  display  numerous  alternative  algorithms.  The  concern  with 
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DSS  is  not  how  it  will  impact  the  decision  itself  but  how  it  will  impact  SSA  subjectivity 
and  the  position  and  power  they  derive  from  this  subjectivity.  Subjectivity  of  the  source 
selection  decision  is  an  important  component  of  the  process  and  the  concern  about 
whether  this  component  can  coexist  with  the  improved  information  provided  by  a  DSS 
must  be  addressed.  The  SSA,  as  the  most  powerful  component  of  the  source  selection 
community,  must  buy-in  to  DSS  implementation  for  it  to  succeed.  Whether  this  can  be 
done  remains  to  be  seen  but  the  possibility  of  eventual  introduction  is  good  when  new, 
less  technology-averse  managers  are  promoted  into  positions  of  power. 

Recommendations 

Further  research  is  certainly  warranted  in  the  area  of  decision  support  systems 
and  their  utilization  in  the  source  selection  process.  The  author’s  recommendations  for 
further  research  in  this  area  are  as  follows: 

—  Conduct  an  experiment  simulating  the  assessment  of  proposals  by  the  technical  and 
contracting  teams  comparing  GDSS  use  with  non-GDSS  use.  Special  attention  should 
be  paid  to  consensus  building  and  its  effect  on  decision  quality,  confidence,  and 
satisfaction. 

—  Conduct  a  survey  of  source  selection  officials  to  determine  their  perceptions 
concerning  GDSS  introduction  at  the  SSET  level  within  the  source  selection  process. 

~  Conduct  an  experiment  with  a  more  complex  scenario  to  further  investigate  the  effect 
of  DSS  on  decision  quality  and  decision  subjectivity 

“  Conduct  a  time  series  analysis  of  DSS  use  to  investigate  the  “learning  cun/e”  effect. 

This  research  revealed  the  improving  perception  of  DSS  technology  within  the 
source  selection  community.  It  also  revealed  that  individuals  involved  with  the  source 
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selection  process  believe  that  DSS  technology  will  be  introduced  into  the  process. 
Clearly,  more  research  is  needed  to  assess  what  type  of  DSS  should  be  implemented, 
what  the  effects  are  of  such  implementation,  and  how  this  technology  can  best  be 
implemented  so  as  to  provide  the  optimal  benefit. 

Conclusions 

The  previous  study  which  addressed  this  issue  (Vickery,  1989)  came  up  with 
many  of  the  same  conclusions  as  this  study.  DSS  has  been  an  available  and  proven 
technology  for  several  years  yet  the  source  selection  community  has  failed  to 
incorporate  this  technology  within  the  process.  If  DSS  technology  is  to  be  implemented 
into  the  source  selection  process,  the  three  characteristics  of  technology  infusion  that 
must  be  addressed  are  compatibility,  complexity,  and  observability  of  outputs.  A 
comprehensive  education  and  training  program  as  well  as  a  pilot  study  can  address 
these  characteristics.  The  training  and  education  of  source  selection  personnel  would 
increase  the  organizational  know-how  with  respect  to  DSS  while  at  the  same  time 
addressing  the  compatibility  and  complexity  characteristics.  The  pilot  study  would  allow 
source  selection  personnel  to  utilize  the  DSS  in  an  actual  source  selection  and  address 
the  observability  of  outputs.  If  the  Air  Force  addresses  these  characteristics  of 
technology  diffusion,  they  will  reap  the  benefits  of  DSS  technology  and  ultimately 
provide  better  products  and  services  to  Air  Force  personnel. 
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Appendix:  Telephone  Interview  Form 


Introduction  (AFIT,  thesis  description,  computer-based  DSS) 
Gather  Background  Data 
Current  position? 

Contracting  experience? 


1 .  What  are  your  present  concerns  with  the  source  selection  process? 

2.  What  can  be  done  to  improve  the  source  selection  process  as  you  see  it? 

3.  How  have  your  feelings/perceptions  about  DSS  changed  during  the  last  few  years 
(5)? 

4.  Has  increased  computer  competence  and  dependence  opened  the  doors  for  DSS 
implementation  in  the  source  selection  process? 

5.  Are  you  presently  using  a  DSS  within  the  source  selection  process? 

Y  —  Where  in  the  process? 

-  Are  you  addressing  other  areas  within  the  process  for  DSS  implementation? 

-  Where  did  you  get  the  DSS?  (developed  yourself,  commercial  off  the  shelf) 

-  What  is  the  impact  of  the  DSS  on  the  source  selection  process? 

N  -  Do  you  foresee  DSS  being  implemented  into  the  source  selection  process? 

Y  -  Where  in  the  process? 

-  Will  it  be  used  by  the  SSA? 

-  Are  you  open  to  the  possibility  of  using  a  DSS  within  your  job? 

Y  -  What  do  you  foresee  its  impact  on  the  process? 

N  -  What  do  you  foresee  being  lot  with  DSS  implementation  in  the 

process? 


6.  How  do  you  believe  the  use  of  a  computer-based  DSS  by  the  SSA  would  impact  the 
following  decision  criteria?  (+,-) 

Decision  time? 

Number  of  alternatives  considered? 

Decision  confidence? 

Decision  satisfaction? 

Decision  quality? 
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3.  Please  estimate  what  this  research  would  have  cost  in  terms  of  manpower  and  dollars  if  it  had 
been  accomplished  under  contract  or  if  it  had  been  done  in-house. 

Man  Years _  $ _ 

4.  Whether  or  not  you  were  able  to  establish  an  equivalent  value  for  this  research  (in  Question 
3),  what  is  your  estimate  of  its  significance? 

a.  Highly  b.  Significant  c.  Slightly  d.  Of  No 

Significant  Significant  Significance 

5.  Comments  (Please  feel  fi-ee  to  use  a  separate  sheet  for  more  detailed  answers  and  include  it 
with  this  form): 


Name  and  Grade 


Organization 


Position  or  Title 


Address 


